Reconstruction for Alexia

I'm back in Berkeley and very grateful to be in the cool weather and sleeping in my own bed. I seriously don't know how East Coast people handle that summer humidity.

I've spent the day getting my reconstruction code into shape for passing it over to Alexia to debug. Some lessons learned today are as follows:

1) I need to stop doing this "copy and paste" thing with python. Time to write up some more functions and get everything in execution mode... Alexia is right, I am wrong.

2) Some plotting tricks:
I've turned my plotting code into some plotting functions (see plotWPS and plotXi functions ../pythonjess/correlationData.py).

These functions now save the plots into the runDirectory as png files. You can look at them over terminal by typing:

> display plotname.png

They look like this (well why are white on black,
I need to figure out how to make them be white-on-black)

If you look at the inner working of these functions I use some commands do the pylab.gcf to set the size and print to a file. I think somewhere in there I can set up a color scheme. Should look into this. I used this web page to help me configure the above plot though.

3) I still don't really understand the different types of ways to write python functions. I've been defining functions as follows:

def function(arguments):
.
.
----Insert function code here
.
.
----return(thing to return)

However it seems like I can also just have a set of code (like a main function in C) that calls a bunch of other functions and isn't a self-contained function... and Demitri said that I just need the following line at the top of the code to make it executable in python:
#!/usr/bin/python

But then at scicoder, Demitri also said that I should create everything as a class. So I guess I just need to figure out what the difference between a class, def, and then just code in a file is.... Maybe Demitri, Alexia, Josh or Adam can help me?

However, as a result of all this effort, I now have code that runs the reconstruction from head-to-tail, and re-runs it once you have already made the correlation functions, and re-runs it with different phi-binning. They are in the following files:

../pythonjess/jessreconstructionFull.py
../pythonjess/jessreconstruction.py
../pythonjess/rerunjessreconstruction.py

These files are the logfiles for today.

Best reconstruction Plot:

=

Time to Get Real

Now that I have a working reconstruction pipeline it is time to run on real SDSS data. I am going to re-run on the data set I have (from CAS database), but I don't believe that the photo-z's are accurate enough to use as the redshifts. To simply use all the spectroscopic data from Sloan isn't going to work because the mask is going to be horribly complicated and I really just want something simple like Stripe 82.

Eyal Kazin has a large catalog of Luminous Red Galaxies that he used to compute the BAO signal. He also has randoms (presumably using a mask) that I could use for calculating the correlation functions. I downloaded this catalog and it only has 105,831 objects in it (3190 in Stripe 82), however the redshift distribution is pretty distinct:

I might be worth trying to reconstruction using his randoms to see what I get. It is probably only 1/2 a day's work to re-write the correlation function to use his randoms instead of generating my own. His random catalog has 15X as many points and has the same redshift distribution (above) and ra, and dec distribution as the data:

The data (blue) falls right on top of the randoms (red)

I have also contacted Eric and Demitri as they have both mentioned having catalog/databases that I might be able to use. I should also contact Antonio.

Python Tip of the Day: I sometimes find that I lose my plotting window. You can close it in the python terminal by the following command: close() and then re-plot to find it again.

Revisiting Likelihood

Nick, David and Adam want me to check that the single epoch likelihood computation is correct. I also need to address some long-standing questions about the likelihood method including:
1. Why are we getting z > 3 when we aren't targeting these objects?
2. What is the proper normalization of the likelihood objects?
And several other questions posted a while back and never answered.

The likelihood code is at the following location on the repository:
https://trac.sdss3.org/browser/repo/boss/bosstarget/trunk/pro/qso-like/

I've also downloaded it here on riemann:
/home/jessica/boss/bosstarget/trunk/pro/qso-like

The varcat files are here (on riemann):
/home/schlegel/varcat4

The star/qso template files are here on the repository:
https://trac.sdss3.org/browser/repo/boss/bosstarget/trunk/data

I've also downloaded it here on riemann:
/home/jessica/boss/bosstarget/trunk/data

Myers sent me a fits file that has the single epoch targets with the likelihoods he computed. He wants me to check that they look reasonable. The file is in an email sent Nov 18/19 from Adam Myers with the subject line: Likelihood Method.

Below is an email I sent to Myers today regarding this check:

-------------------
From: Jessica Ann Kirkpatrick
Date: Mon, Nov 23, 2009 at 4:15 PM
Subject: Re: Likelihood method
To: Nic Ross
Cc: Joseph Hennawi , Adam Myers, David Schlegel

Adam,
when I compute the likelihoods I get the same values as you do for
like_ratio, like_everything, like_QSO_Z (well, within expected
rounding errors).

One thing that I had a question about is how like_rank is numbered. I
was assuming that like_rank = 0 would be the target with the largest
like_ratio, but it seems it is the reverse (like_rank = 0 is the
lowest like_ratio object).

So we are sorting target selection by largest ranking to smallest?

I've attached a color-color (ug, gr) plot of the 10,000 highest ranked
likelihood objects (cyan) on top of "everything" objects (red) and
qsos (green). You can see that these targets fall in the quasar zone.

Color-Color (ug, gr) plot of the 10,000 highest ranked
likelihood objects (cyan) on top of "everything" objects (red) and
qsos (green). You can see that these targets fall in the quasar zone.

Useful Mac Tips of the Day (via Demitri):
If you want to ssh into your Mac, go to

• System Preferences -> Sharing -> Remote Login
• Turn Remote Login on. Then find out your IP address by typing the following into a web browser: http://whatismyipaddress.com/

You can then ssh/scp into your Mac in the normal way using jessica@myipaddress!

Also... Alexia will be happy to know that I finally implemented different terminal colors for different computers that I am logged into (so much less confusing):

Top terminal window (green) is riemann,
lower terminal window (black) is my Macbook.

And... Demitri showed me how to turn any mp3 into a ring tone using Garage Band. What a great day for cool tricks! Now my alarm clock will be the beginning of James Brown's Sex Machine: "Get-up, get-on-up. Get-up, get-on-up."

I am HUGE in Berkeley (and NY)

Blogger.com was down last Thursday and not allowing me to post. Sorry for all of you who were eagerly awaiting my blog postings and disappointed by me not fulfilling my quota last week. Apparently my not posting (combined with not returning phone calls) made my father think I had died. Sorry Dad!

Below is the belated post for Thursday:
I re-ran the 3D auto-correlation function code with courser binning as suggested by Alexia. My result is not that different, although I did figure out how to put axes labels and a legend on my plots, so all was not lost!



The numbers in the legend represent the lower edge of the line-of-sight bin of the spectroscopic data in the 3D auto-correlation function in Mpc. So for instance the yellow line is a bin from 0-200 Mpc, the redline is from 600-800 Mpc etc.

I installed Google Analytics on this blog. Apparently my fan base is mostly Berkeley (5 visits -- probably mostly myself?) followed by New York (3 visits -- Demetri? Kathryn?). Unfortunately no visits from Princeton which means Alexia isn't reading this. Ha ha, busted!

Next step is to try calculating the 3D correlation function on a fake data set and comparing it to the known answer to make sure this isn't a bug in my code.