Why do I feel intense anxiety before I have a let down and sometimes while nursing my baby?

I am a little concerned because with my son I am having these intense feelings of anxiety right before let down. It's almost like a panic attack. It doesn't seem to affect my let down, but why do I feel this way with him? I really love and adore my son and I wouldn't stop feeding him for anything, but why am I feeling this way? When I nursed my last child I felt nothing but intense feelings of love and happiness before a let down. Now I'm anxious and depressed right before a let down. My grandmother died suddenly while i was pregnant and I was very close to her, could this have some sort of subconscious effect on my hormones or something? Any advice is appreciated.