Reconstruction on LRGs

I talked to Alexia today and we decided that I should get the BOSS data together and run correlation functions on it, rather than continuing to de-bug the reconstruction binning. I'm a little frustrated with that problem, and it would be nice to work on something new. I know that both Shirley Ho and Eyal Kazin have LRG catalogs, so I am contacting both of them to see if they are available for me to use.

Also planning to visit Alexia in late October/ early November. Trying to find cheap housing in Los Alamos, if anyone knows of anything out there...

1M Reconstruction

I ran a big reconstruction on the CAS data using 1,000,000 objects, and 20 redshift bins. The correlation functions at the lower redshifts (less than 720 Mpc/h) have much less noise:


But the correlation functions at larger redshifts look pretty noisy:

Perhaps I should bin so that I have a constant number of objects per bin instead of by equal redshift bins.

When I do the reconstruction, I get puzzling results:

The reconstruction seems to think there are a large number of points at redshift > 1.0 which aren't there. This could be due to low statistics at these redshifts and so the reconstruction just isn't working properly. The number of spectroscopic objects per bin is as follows:

0-120 Mpc/h: 14597 galaxies in spectroscopic dataset
120-240 Mpc/h: 45954 galaxies in spectroscopic dataset
240-360 Mpc/h: 55489 galaxies in spectroscopic dataset
360-480 Mpc/h: 59451 galaxies in spectroscopic dataset
480-600 Mpc/h: 64381 galaxies in spectroscopic dataset
600-720 Mpc/h: 56290 galaxies in spectroscopic dataset
720-840 Mpc/h: 55916 galaxies in spectroscopic dataset
840-960 Mpc/h: 49536 galaxies in spectroscopic dataset
960-1080 Mpc/h: 29109 galaxies in spectroscopic dataset
1080-1200 Mpc/h: 20852 galaxies in spectroscopic dataset
1200-1320 Mpc/h: 16729 galaxies in spectroscopic dataset
1320-1440 Mpc/h: 13715 galaxies in spectroscopic dataset
1440-1560 Mpc/h: 8969 galaxies in spectroscopic dataset
1560-1680 Mpc/h: 5094 galaxies in spectroscopic dataset
1680-1800 Mpc/h: 2374 galaxies in spectroscopic dataset
1800-1920 Mpc/h: 890 galaxies in spectroscopic dataset
1920-2040 Mpc/h: 381 galaxies in spectroscopic dataset
2040-2160 Mpc/h: 169 galaxies in spectroscopic dataset
2160-2280 Mpc/h: 85 galaxies in spectroscopic dataset
2280-2400 Mpc/h: 16 galaxies in spectroscopic dataset

It looks like I shouldn't use bins greater than ~1Mpc/h for the reconstruction because we have so few galaxies out there. If I try doing the reconstruction only using bins up to 1080 Mpc/h this is that I get:

There seems to be a problem at the earlier redshifts, perhaps due to low statistics in the first redshift bin? But the reconstruction looks sort of good.

I'm need to see what Alexia thinks about this "forcing to zero" issue at the edges. I also want to try this with the LRG catalog. I'm going to set a run going where I bin such that there are constant numbers of galaxies per bin, not constant redshifts. Alexia, let's chat tomorrow early?

Time to Get Real

Now that I have a working reconstruction pipeline it is time to run on real SDSS data. I am going to re-run on the data set I have (from CAS database), but I don't believe that the photo-z's are accurate enough to use as the redshifts. To simply use all the spectroscopic data from Sloan isn't going to work because the mask is going to be horribly complicated and I really just want something simple like Stripe 82.

Eyal Kazin has a large catalog of Luminous Red Galaxies that he used to compute the BAO signal. He also has randoms (presumably using a mask) that I could use for calculating the correlation functions. I downloaded this catalog and it only has 105,831 objects in it (3190 in Stripe 82), however the redshift distribution is pretty distinct:

I might be worth trying to reconstruction using his randoms to see what I get. It is probably only 1/2 a day's work to re-write the correlation function to use his randoms instead of generating my own. His random catalog has 15X as many points and has the same redshift distribution (above) and ra, and dec distribution as the data:

The data (blue) falls right on top of the randoms (red)

I have also contacted Eric and Demitri as they have both mentioned having catalog/databases that I might be able to use. I should also contact Antonio.

Python Tip of the Day: I sometimes find that I lose my plotting window. You can close it in the python terminal by the following command: close() and then re-plot to find it again.

Notes from the Beach

I have spent the last two weeks at two conferences. First the American Astronomical Society (AAS) 215th Meeting in Washington DC, second Essential Cosmology for the Next Generation: Cosmology on the Beach in Playa del Carmen, Mexico. Both were really helpful for both learning new astrophysics, as well as getting lots of new ideas for my work.

I thought I would summarize here the highlights (with links to talks, and a list of things I would like to look into myself).

Max Tegmark coined the phrase "Goldilocks Galaxies" when talking about the Luminous Red Galaxies (LRGs) in SDSS. This is because "Quasars are too sparse, common galaxies are too dense, but LRGs are just right."

Mark also talked about 21 cm tomography comparing it to the CMB in it's potential importance to cosmology. His description unfortunately was pretty general and I don't think I understand how it works. Need to read a review paper on this. Looks like this might be a good one.

Nicholas Suntzeff had an interesting slide at the end of his talk where he gave advice to young cosmologists about how to be competitive for jobs. I've repeated it below:

• Don’t keep on doing your thesis over and over again
• Establish prominent collaborators and mentors, but appear independent
• Publish, publish, publish. Include useful tables of summary and colorful figures that can be easily captured.
• Apply for external funding
• Luck verus hard work
• Become the leader in your field
• Think carefully about joining large projects with time scales of > 5 years.
• Spergel’s law
• Don’t be afraid to go out on a limb and say something weird (people will remember you).
• The Aaronson effect in obsevations.
• When you apply for jobs, make sure you know all about the department – and brown nose a bit. Write your application as if there is no other job out there. Know your audience.
  

Berian James is a new post-doc at the BCCP starting this spring. He had some ideas about Newman project and had talked to others who are working on similar things (he named John Peacock and Hannah Parkinson at Edinburgh). He also had an interesting idea that it would be great if we could positions from photometry and a redshift distribution and generate a correlation function from these two things. He was wondering if anyone has done this. I should talk to him more about this when he gets to Berkeley.

Dovi Poznanski suggested I look at quasar variability using overlapping Sloan plates. Apparently 20% of the data has multiple epochs. Josh Bloom also told us that we could get good variability information using Palomar data. Nic Ross and I should set up a meeting with him when we get back.

Daniel Matthews is Jeff Newman's graduate student. He presented a poster at the AAS (see Jan 2nd email, subject: poster, he sent me a copy). In this poster he talked about doing redshift distribution reconstructions of "DEEP-like" data. He does something different than what Newman does in his paper. He claimed that he couldn't get Newman's method to work -- this is discouraging. Maybe it's time to try to talk to Newman directly about the project to clarify this?

The Joe Wolf Effect.

Marilena LoVerde discussed lensing of quasars from hydrogen in the ly-alpha forest. I talked to her afterwards and it doesn't seem like this effect his that large, but might be relevant for BOSS/BIG BOSS. The talk isn't posted on the Cosmology on the Beach web page as of right now.

Eyal Kazin has a web page with LRG catalogs. He said he is interested in the Newman Project and is very familiar with SDSS data, and to contact him if we want help. These LRGs might be useful data to use instead of the ones I downloaded from CAS, and don't really trust.

I had some thoughts about if I should request authorship on future DRIFT analysis papers, considering they didn't allow me to publish my work. I don't know what the protocol for this type of thing is, or if it even matters at this point in my career.

I had a great time at these conferences and learned a lot. Time to get back to real work now! Science to follow....

Catalog Woes

It turns out my mock catalog is probably not what I want to be using. I talked to Alexia and Martin White, and I am using a LRG catalog, but I want to be using a halo catalog. This probably means that my clustering properties are nonsensical (because I am generating galaxies based on galaxies, not on halos). Martin said he is going to give me a better catalog to use.

In the mean time I am running on the catalog I have. The reconstruction should work as long as both the photometric and spectroscopic data sets are generated in the same way... even if they don't accurately represent any real distribution on the sky.

The reconstruction is currently running, and I should have it done later today.

I also checked my 3D slice against Alexia's to make sure they match, and they do:

My and Alexia's 3D Correlation functions on
a redshift slice of the mock data

Talking with Weinberg

I ran into David Weinberg at tea at the IAS today. We started talking about the Newman Project.
He suggested that after we get the method working on Stripe 82 using the LRGs as our "spectroscopic sample" and the rest of the galaxies as our "photometric sample" we could compare the reconstructed distribution with spectroscopic "follow-up" surveys such as COSMOS, Vimas?, and DEEP.

Then it would be interesting to break down the main sample (i.e. the rest of the galaxies) by colors and do redshift distribution reconstruction on each type of galaxy. We could also break these into photometric redshift slices and then look at the reconstruction in each slice and break those down by colors, to see if perhaps there is a different spread in the colors.

I thought these were really good ideas and ones that I haven't though of before. Need to talk to Schlegel/Nikhil and see what they think. Oh exciting!

Coordinate Confusion

Time to run my 3D autocorrelation function on a mock catalog where we know the answer to try to see if there is a problem with my code or that the reconstruction is failing for another reason. I downloaded the following raw mock LRG catalog from Martin White's web page: halo_000_0.8000.dat.gz. I discovered that I can transfer files to riemann via the insure replacement by doing a two step scp. That is useful to know.

The code I am using to convert from ra, dec (degrees), comoving distance (r) to x, y, z and back:

%  x=r*sin(pi/2-pi/180*dec)*cos(pi/180*ra);
%  y=r*sin(pi/2-pi/180*dec)*sin(pi/180*ra);
%  z=r*cos(pi/2-pi/180*dec) ;

%  r = sqrt(x^2 + y^2 + z^2)
%  dec =  90 - 180/pi*arccos(z/r)
%  ra = 180/pi*arcsin(y/sqrt(x^2 + y^2))

I am a little confused because these mock files are in Cartesian coordinates and if I convert them using the above code into ra and dec, I don't get the objects populating all of ra/dec space:

I constrained the data to be in a sphere of radius 0.5 boxsize (936.0 Mpc/h), and I would think that the ra and dec would then go from 0 to 360 and 0 to 180 respectively.

After some digging, I discovered that by using the arctan2 function I get the proper range:

%  theta = arctan2(sqrt(x^2+y^2), z)
%  phi = arctan2(x, y)

However when I try to then convert back to Cartesian I get a strange answer when I plot the original z versus the converted back z:

This is distressing!

Sources:
http://astro.uchicago.edu/cosmus/tech/code/radecz2xxyyzz.m
http://www.atlasoftheuniverse.com/cosmodis.c
http://www.math.montana.edu/frankw/ccp/multiworld/multipleIVP/spherical/body.htm
http://www.daniweb.com/forums/post860497-2.html

Python tips for the day
1) If you are getting the following error when trying to plot:

RuntimeError: Agg rendering complexity exceeded.
Consider downsampling or decimating your data

Then exit and restart your session and try again and it for some reason works. Stupid python.

2) When converting from spherical use atan2 not asin.