Tuesday, September 20, 2016

formatting SNPs using R or awk

In some software for genomic prediction (blupf90, GS3 and may be other) the genotypes shoukd be given in a plain text file as follows:

snp_file.txt

          25 1121022100
         600 0111220012
        1333 0110111111
           5 1120112102
          89 0111220001 


with no spaces between genotypes, id and genotypes separated by spaces - no tabs - and all genotypes starting at the same column. Sometimes it is not obvious how to get this format. Llibertat Tusell got a solution in R:

snps=sample(c(0,1,2),prob=c(.25,0.5,.25),size=50,replace=T)
X=matrix(ncol=10,nrow=5,snps)
animal=c(25,600,1333,5,89)

con <- file("snp_file.txt", "w")

for (i in 1:5){
  tmp=paste( X[i,] , collapse = "" )
  cat(format(animal[i],width=12),tmp,'\n',file=con)
}
close(con)


The R program collapses markers into a single string, then puts format to the animal "word" so that it has constant width, then it writes it to a file

Another solution is to use awk and start from a file, e.g.

$ cat exo_geno_spaces 
  1101 1 0 2 2 1 1 2
     101 2 2 1 1 2 1 1
   254 1 1 2 0 1 1 0

   255    2 1 2 2 0 1 2

Then you can use an awk program ./remove_spaces_snps.awk:

#! /opt/local/bin/gawk -f
# this script removes space between SNP genotypes
# and formatting as UGA
BEGIN{}
{
    # print animal
    printf( "%20s",$1)
    printf( "%1s"," ")

    for (i =2; i<=NF; i++){
      printf( "%1s",$i)
    }
    printf("\n")
 }
END{}

This awk program prints on stdout the animal (with constant width) then markers without sopace separation, then a newline.

$ ./remove_spaces_snps.awk exo_geno_spaces > out
$ cat out 

                1101 1022112
                 101 2211211
                 254 1120110

                 255 2122012

Monday, September 19, 2016

Livestock Fair: Pirenaica cattle

I was in a livestock fair in Irurtzun and I could take good pictures of local breeds. These are Pirenaica cows and calves. The Pirenaica breeding association is Conaspi and is has been the object of many scientific publications.





In the same village, there used to be a weekly livestock fair every tuesday until the 70's. In this old picture circa 1950 you can see some local animals (some of them Pirenaica) that were used, among other things, for working the fields. 






Thursday, September 15, 2016

Reordering matrix in R

I have this matrix with relationships across metafounders and years. However the matrix is disordered, let's say:

> g
     1998 2010 1970
1998 0.90 0.67 0.83
2010 0.67 0.92 0.52
1970 0.83 0.52 0.95

I would like to sort this matrix in ascending order, i.e. the cell corrresponding to [1970,1970] should go on top left.

This can be done using a loop but it is tricky. In R this is easy using sorted indices:

> g1=g[order(rownames(g)),order(colnames(g))]
> g1
     1970 1998 2010
1970 0.95 0.83 0.52
1998 0.83 0.90 0.67
2010 0.52 0.67 0.92



Tuesday, September 13, 2016

ifort options -openmp and -qopenmp

I work in three computers, two servers (let's call them grits and cassoulet) and my own Mac machine. Well, I found out that the options to use OpenMP in the Intel Fortran compiler (ifort) are different depending on the compiler version:

  • grits: ifort 15.0.3: -qopenmp
  • Mac: ifort 15.0.2: -qopenmp
but...

  • cassoulet: ifort 14.0.3: -openmp
According to the documentation for for ifort, the option -openmp is deprecated .


Friday, July 29, 2016

gawk in MobaXterm

MobaXterm is a nice unix/linux console emulator. We were trying to use it to run a script for genetic evaluation, yet in one of the awk scripts we used the sentence
 BEGIN {FIELDWIDTHS = "12 14 8 2 3 3 8 1 3 14 14 3 8 14" }

it seems that this FIELDWIDTHS is restricted to gawk, but not to all implementations of awk, in particular not to the one in MobaXterm. However MobaXterm has the possibility of installing programs and plugins. Thus I try

[andres.ANDRESLEGAR2422] → apt-get
apt-cyg: Installs and removes Cygwin packages.
  "apt-cyg install <package names>" to install packages

So, I install it

[andres.ANDRESLEGAR2422] → apt-get install gawk

Trying to download file setup.bz2
Updated setup.ini
Found package gawk
Downloading gawk-4.1.3-1.tar.xz...
Unpacking gawk-4.1.3-1.tar.xz...
Extracting dependencies for usr/bin/gawk.exe...
Extracting dependencies for usr/bin/gawk-4.1.3.exe...
Extracting dependencies for usr/libexec/awk/pwcat.exe...
Extracting dependencies for usr/libexec/awk/grcat.exe...
Extracting dependencies for usr/lib/gawk/filefuncs.dll...
Extracting dependencies for usr/lib/gawk/fnmatch.dll...
Extracting dependencies for usr/lib/gawk/fork.dll...
Extracting dependencies for usr/lib/gawk/inplace.dll...
Extracting dependencies for usr/lib/gawk/ordchr.dll...
Extracting dependencies for usr/lib/gawk/readdir.dll...
Extracting dependencies for usr/lib/gawk/readfile.dll...
Extracting dependencies for usr/lib/gawk/revoutput.dll...
Extracting dependencies for usr/lib/gawk/revtwoway.dll...
Extracting dependencies for usr/lib/gawk/rwarray.dll...
Extracting dependencies for usr/lib/gawk/testext.dll...
Extracting dependencies for usr/lib/gawk/time.dll...
Package gawk requires the following packages, installing bash cygwin libgcc1 libgmp10 libintl8 libmpfr4 libreadline7
Package bash is already installed, skipping
Package cygwin is already installed, skipping
Found package libgcc1
Package libgcc1 is already included, skipping
Found package libgmp10

Installing libgmp10
Downloading libgmp10-6.1.0-3p1.tar.xz...
Unpacking libgmp10-6.1.0-3p1.tar.xz...
Extracting dependencies for usr/bin/cyggmp-10.dll...
Package libgmp10 requires the following packages, installing cygwin
Package cygwin is already installed, skipping
Package libgmp10 installed.
Found package libintl8
Package libintl8 is already included, skipping
Found package libmpfr4

Installing libmpfr4
Downloading libmpfr4-3.1.4-1.tar.xz...
Unpacking libmpfr4-3.1.4-1.tar.xz...
Extracting dependencies for usr/bin/cygmpfr-4.dll...
Package libmpfr4 requires the following packages, installing cygwin libgcc1 libgmp10
Package cygwin is already installed, skipping
Found package libgcc1
Package libgcc1 is already included, skipping
Package libgmp10 is already installed, skipping
Package libmpfr4 installed.
Found package libreadline7
Package libreadline7 is already included, skipping
Package gawk installed.

Rebasing new libraries

Found package rebase

Installing rebase
Downloading rebase-4.4.2-1.tar.xz...
Unpacking rebase-4.4.2-1.tar.xz...
Extracting dependencies for usr/bin/rebase.exe...
Extracting dependencies for usr/bin/peflags.exe...
Package rebase requires the following packages, installing coreutils cygwin grep gzip sed
Found package coreutils
Package coreutils is already included, skipping
Package cygwin is already installed, skipping
Package grep is already installed, skipping
Found package gzip
Package gzip is already included, skipping
Found package sed
Package sed is already included, skipping
Package rebase installed.

Then I change in the scripts to use /bin/gawk.exe 
Now everything seems to work fine.


Wednesday, July 13, 2016

"new" equation editor in word

Since circa 2010 you can do nice things with the Equation Editor  (even for Mac) in Word and Power Point. For instance my notes on genomic prediction. Note that this is NOT the horrible Equation Editor that you think:


But a new one which has LaTex-y kind of input, using Math Autocorrect:

which is based on this. It works nicely in PowerPoint too:



 So, I completely gave up using MathType.

However, there are no good resources for this nrew Equation Editor, and the web of Math Autocorrect is not very good.

I used this documentati to learn (although it is a bit outdated):
http://school-maths.com/documents/Word_2010_stols.pdf .

You may also like these two titles:

The Word 2007/2010 Equation Editor
Using Keystrokes to Write EquationsIn Microsoft Office 2007 Equation Editor 


A cheatsheet with shortcuts.

There is also MathInOffice is a very nice resource but not exactly a tutorial.

Thursday, July 7, 2016

Working in Windows machines

To my taste, Windows is inefficient for serious number crunching. But if you need to do so, and you feel like doing things the Unix way, you may:


  1. install MobaXterm, a nice unix terminal emulator
  2. install notepad++, en excellent plain text editor
  3. get a linux/unix command cheatsheet (most commands are in MobaXterm):
    1. In Spanish: Comandos Unix/Linux – Guía de Referencia
    2. In French
    3. or in English
  4. I also recommend to use awk (although this is by no means absolute need). For an awk manual, the best one that I know is in Spanish, but there are many out there.
  5. And finally, R and gfortran .

There is always the question on how to open a terminal. Here it goes for Windows 10 and for Windows 7.