Wednesday, January 25, 2017

transposing marker data

Quite frequently one may find marker data ordered in this way:

alegarra@genotoul2 ~/save/progs $ cat ex_rachel
 1 a b
 1 b b
 1 c c
 2 b b
 2 c c
 2 d d
 3 a b
 3 b b
 3 c c

e.g. there are three animals (1 to 3) and three markers. Rachel and I would like them to be formatted one animal per line and markers one after each other, in this way:


alegarra@genotoul2 ~/save/progs $ cat out
         1 ab bb cc 
         2 bb cc dd 
         3 ab bb cc 

This is conceptually simple if animals are sorted:
  1. Read a line
  2. If the animal is the same as the old one, print the markers after the previous one
  3. If the animal is different, start a new line, print the animal and the markers.
  4. Add special cases for the first and last line
Here is an awk implementation


alegarra@genotoul2 ~/save/progs $ cat SNPcol2line.awk 
#! /bin/awk -f
# this program reads genotypes in one line per locus
# then puts them as
# individual allele1 allele2 allele1 allele 2
#
BEGIN{
idold=0
i=0
}
{
id=$1
# if new animal
if(id!=idold){
if(idold!=0){
# close previous line
printf("\n")
}
# write new id
printf("%10s%1s",id," ")
idold=id
}
printf("%1s%1s%1s",$2,$3," ")
}
END{
# last individual
printf("\n")
}

which works:

alegarra@genotoul2 ~/save/progs $ ./SNPcol2line.awk ex_rachel 
         1 ab bb cc 
         2 bb cc dd 
         3 ab bb cc 

Friday, December 9, 2016

Sires in validation and Sires in training

I want to test metafounders for genomic selection. So I have these MTR sires that have daughters in the validation , in the training, or in both of them, and I want a file with these numbers. I have two files:

T is:
juan 10
pepe 5


V is:

luis 6
pepe 4


so luis has no daughters on Training, and I want this to appear. Then I use this pipeline from Toni Reverter from CSIRO:

awk '{print $1}' $1 $2 | sort -u | join -a1 - $1 | awk '{print $1, (NF==2?$2:0)}' | join -a1 - $2 | awk '{print $1, $2, (NF==3?$3:0)}'


which gives

putterri:yarp andres$ ./joinToni.sh V T
juan 0 10
luis 6 0

pepe 4 5





Install GNU coreutils in the Mac

The MacOSX join is poorer than GNU one. So I try toinstall it using macports. First I have to update the data base of port:

putterri:mf andres$ port -v selfupdate

But this does not work. Then I try


putterri:mf andres$ sudo port -d selfupdate

This does work. Join is actually part of coreutils:


putterri:mf andres$ port search --name coreutils
coreutils @8.25 (sysutils)
    GNU File, Shell, and Text utilities

xml-coreutils @0.8.1_1 (textproc, xml)
    Command line tools for XML processing

Found 2 ports.

So I do install coreutils:


putterri:mf andres$ sudo port install coreutils

The most important information is at the end:

The tools provided by GNU coreutils are prefixed with the character 'g' by default to distinguish them from the BSD commands.
For example, cp becomes gcp and ls becomes gls.

If you want to use the GNU tools by default, add this directory to the front of your PATH environment variable:
    /opt/local/libexec/gnubin/


Saturday, November 19, 2016

Plotting correlations

I found this nice R package to plot correlations : corrplot . Imagine that you want to present five genetic correlations to your buddies. Here's how to visualize them nicely.

#create a covariance matrix
set.seed(1234)
V=rWishart(1,10,diag(5))[,,1]
cov2cor(V)
            [,1]       [,2]       [,3]       [,4]        [,5]
[1,]  1.00000000  0.1169636  0.2236465 -0.2667448  0.04351447
[2,]  0.11696358  1.0000000 -0.2260988 -0.3425368  0.64837326
[3,]  0.22364648 -0.2260988  1.0000000 -0.1883763 -0.22484960
[4,] -0.26674477 -0.3425368 -0.1883763  1.0000000 -0.50315007
[5,]  0.04351447  0.6483733 -0.2248496 -0.5031501  1.00000000
#give it names
colnames(V)=c("MY","gain","longevity","SCS","pietin")
rownames(V)=c("MY","gain","longevity","SCS","pietin")

#plot
require(corrplot)
corrplot.mixed(cor(V),col=gray.colors(10))


Thursday, October 27, 2016

French Keyboard on US Macbook Keyboard

I have a US-like keyboard with this key disposition


However, it happens to me to write in French changing keyboard preferences to French. The keys are mapped to a French keyboard.  I use a rubber cover from kbcovers.com, which is excellent and looks like this:


However, the physical layout of a true French macbok keyboard is slightly different. Also, and more impostant, some important characters such as | ~ # [ ] { } are not present, or difficult to see, in the cover.

So, I prepared my own cheatsheet:


Yes, it does look horrible and yes, it does work :-)

Tuesday, September 20, 2016

formatting SNPs using R or awk

In some software for genomic prediction (blupf90, GS3 and may be other) the genotypes shoukd be given in a plain text file as follows:

snp_file.txt

          25 1121022100
         600 0111220012
        1333 0110111111
           5 1120112102
          89 0111220001 


with no spaces between genotypes, id and genotypes separated by spaces - no tabs - and all genotypes starting at the same column. Sometimes it is not obvious how to get this format. Llibertat Tusell got a solution in R:

snps=sample(c(0,1,2),prob=c(.25,0.5,.25),size=50,replace=T)
X=matrix(ncol=10,nrow=5,snps)
animal=c(25,600,1333,5,89)

con <- file("snp_file.txt", "w")

for (i in 1:5){
  tmp=paste( X[i,] , collapse = "" )
  cat(format(animal[i],width=12),tmp,'\n',file=con)
}
close(con)


The R program collapses markers into a single string, then puts format to the animal "word" so that it has constant width, then it writes it to a file

Another solution is to use awk and start from a file, e.g.

$ cat exo_geno_spaces 
  1101 1 0 2 2 1 1 2
     101 2 2 1 1 2 1 1
   254 1 1 2 0 1 1 0

   255    2 1 2 2 0 1 2

Then you can use an awk program ./remove_spaces_snps.awk:

#! /opt/local/bin/gawk -f
# this script removes space between SNP genotypes
# and formatting as UGA
BEGIN{}
{
    # print animal
    printf( "%20s",$1)
    printf( "%1s"," ")

    for (i =2; i<=NF; i++){
      printf( "%1s",$i)
    }
    printf("\n")
 }
END{}

This awk program prints on stdout the animal (with constant width) then markers without sopace separation, then a newline.

$ ./remove_spaces_snps.awk exo_geno_spaces > out
$ cat out 

                1101 1022112
                 101 2211211
                 254 1120110

                 255 2122012

Monday, September 19, 2016

Livestock Fair: Pirenaica cattle

I was in a livestock fair in Irurtzun and I could take good pictures of local breeds. These are Pirenaica cows and calves. The Pirenaica breeding association is Conaspi and is has been the object of many scientific publications.





In the same village, there used to be a weekly livestock fair every tuesday until the 70's. In this old picture circa 1950 you can see some local animals (some of them Pirenaica) that were used, among other things, for working the fields.