Showing posts with label CAS. Show all posts
Showing posts with label CAS. Show all posts

Tuesday, February 2, 2010

1M Reconstruction

I ran a big reconstruction on the CAS data using 1,000,000 objects, and 20 redshift bins. The correlation functions at the lower redshifts (less than 720 Mpc/h) have much less noise:


But the correlation functions at larger redshifts look pretty noisy:


Perhaps I should bin so that I have a constant number of objects per bin instead of by equal redshift bins.

When I do the reconstruction, I get puzzling results:


The reconstruction seems to think there are a large number of points at redshift > 1.0 which aren't there. This could be due to low statistics at these redshifts and so the reconstruction just isn't working properly. The number of spectroscopic objects per bin is as follows:

0-120 Mpc/h: 14597 galaxies in spectroscopic dataset
120-240 Mpc/h: 45954 galaxies in spectroscopic dataset
240-360 Mpc/h: 55489 galaxies in spectroscopic dataset
360-480 Mpc/h: 59451 galaxies in spectroscopic dataset
480-600 Mpc/h: 64381 galaxies in spectroscopic dataset
600-720 Mpc/h: 56290 galaxies in spectroscopic dataset
720-840 Mpc/h: 55916 galaxies in spectroscopic dataset
840-960 Mpc/h: 49536 galaxies in spectroscopic dataset
960-1080 Mpc/h: 29109 galaxies in spectroscopic dataset
1080-1200 Mpc/h: 20852 galaxies in spectroscopic dataset
1200-1320 Mpc/h: 16729 galaxies in spectroscopic dataset
1320-1440 Mpc/h: 13715 galaxies in spectroscopic dataset
1440-1560 Mpc/h: 8969 galaxies in spectroscopic dataset
1560-1680 Mpc/h: 5094 galaxies in spectroscopic dataset
1680-1800 Mpc/h: 2374 galaxies in spectroscopic dataset
1800-1920 Mpc/h: 890 galaxies in spectroscopic dataset
1920-2040 Mpc/h: 381 galaxies in spectroscopic dataset
2040-2160 Mpc/h: 169 galaxies in spectroscopic dataset
2160-2280 Mpc/h: 85 galaxies in spectroscopic dataset
2280-2400 Mpc/h: 16 galaxies in spectroscopic dataset

It looks like I shouldn't use bins greater than ~1Mpc/h for the reconstruction because we have so few galaxies out there. If I try doing the reconstruction only using bins up to 1080 Mpc/h this is that I get:

There seems to be a problem at the earlier redshifts, perhaps due to low statistics in the first redshift bin? But the reconstruction looks sort of good.

I'm need to see what Alexia thinks about this "forcing to zero" issue at the edges. I also want to try this with the LRG catalog. I'm going to set a run going where I bin such that there are constant numbers of galaxies per bin, not constant redshifts. Alexia, let's chat tomorrow early?

Thursday, January 28, 2010

Working Data Reconstruction!

I ran the SDSS CAS data though our working pipeline. I've run on a small data set as a first go -- the same size as what I was running on for the reconstructions with the mock data -- 20,000 points in both the photometric and spectroscopic data sets. The difference here is that both data sets are coming from the same super-set of data (I am randomly picking points from a larger set of Stripe-82 data), so the redshift distributions of the two sets are the same (see below plot). This, in theory, should make the reconstruction easier to do.

Redshift Distributions of photometric and spectroscopic data sets


The Bad News
The correlation functions look really noisy. This is hopefully due to the fact that I am running with 20,000 points instead of 1,000,000 points (before) so hopefully the noise will clear up when I re-run with a larger data set. I wouldn't expect the correlations to look as smooth on a smaller data set as it does with the mock data, because the mock data, by design, traces dark matter populations explicitly, so the dark matter signal is put in by hand. There is also more noise on this data than on the mock data. It is logical to me that we might need higher statistics to get the same strength of signal as we did in the mock data.


The Good News
Even with the noisy correlation functions, the reconstruction looks not too bad. There seems to be a problem at the edges (it seems to be forcing the reconstruction to be zero at redshift zero and thus changing the shape on the low end), but overall it looks pretty good. (At least much better than it's ever looked before). Thoughts Alexia?


I've chosen the binning which looks the best. Like what happened in the mock data the reconstruction changes a lot depending on the binning. However, it looks like I might actually have something working here! Woo hoo!

Next steps
Higher Statistics: I'm not sure if I should run on Eyal's LRGs or on a bigger set of CAS data. The CAS data is easier to do right away, so I might as well set that going (it will take a couple days), and in the mean time I can work on getting my code to read in Eyal's randoms instead of generating it's own. Will talk to Schlegel about this and see what he thinks.

Other Catalogs: According to Eric Huff, Reiko Nakajima has a SDSS catalog with her own calculated photo-z's that she uses for lensing. I will talk to her about this. I also should get in touch with Antonio Vazquez about catalogs.

Optimization: There is quite a few things I can do to make this code run faster. I need to spend some time writing scripts to simultaneously calculate all the correlation functions (right now they are calculated in succession). This should speed up the code by a factor of ten. I also should write scripts so that I can just set a run going and then log-out of the terminal window.

Organizational Note
For a while now I've been creating a log file for each day with the code I use to generate the plots for the blog post that day. I've been keeping these log files locally on my computer, but decided it might be useful for collaborators to have copies of them, and also for there to be a back-up should something happen to my laptop. I've now added these logs to the repository under: repositoryDir/Jessica/pythonJess/logs. The name of the log file is the date of the blog post which the code corresponds to.

Monday, January 25, 2010

Time to Get Real

Now that I have a working reconstruction pipeline it is time to run on real SDSS data. I am going to re-run on the data set I have (from CAS database), but I don't believe that the photo-z's are accurate enough to use as the redshifts. To simply use all the spectroscopic data from Sloan isn't going to work because the mask is going to be horribly complicated and I really just want something simple like Stripe 82.

Eyal Kazin has a large catalog of Luminous Red Galaxies that he used to compute the BAO signal. He also has randoms (presumably using a mask) that I could use for calculating the correlation functions. I downloaded this catalog and it only has 105,831 objects in it (3190 in Stripe 82), however the redshift distribution is pretty distinct:


I might be worth trying to reconstruction using his randoms to see what I get. It is probably only 1/2 a day's work to re-write the correlation function to use his randoms instead of generating my own. His random catalog has 15X as many points and has the same redshift distribution (above) and ra, and dec distribution as the data:

The data (blue) falls right on top of the randoms (red)

I have also contacted Eric and Demitri as they have both mentioned having catalog/databases that I might be able to use. I should also contact Antonio.

Python Tip of the Day: I sometimes find that I lose my plotting window. You can close it in the python terminal by the following command: close() and then re-plot to find it again.