I just went and looked at that center piece in my car. I don't have the speaker, but the process should be the same. Mine's a 1995. Not that much difference other than the panel has a speaker cover (like yours) or doesn't.
(like mine)
So, this is just based on what I saw in my car and what I read about removing the "rear console trim" in the manual....so, no promises, but you can try and see what happens.
The rear, passenger side seatback flips down for a pass-thru to the trunk, so you can put long objects in the trunk.
There is a key lock on the top of the trunk deck shelf that releases the seatback. Use your ignition key to unlock the lock and flip the seat down.
Now, with the seat down take a look at the bottom side of that speaker center panel. There should be a plastic pop-out plug down there. And if you push the seat back foam in with your hand on the other rear seat, you should be able to access that same plug on the other side.
Remove those two plastic anchor/plugs and then lift that middle panel up and off. BUT be careful, because, according to the shop manual diagram, there are two plastic trim locks at each of the top corners of that piece and they hold the top of the panel in a locked down position so it remains flush with the trunk deck. DON'T BREAK THEM OFF by forcing it up or out.
I don't know if the panel slides directly up, lifts out at an angle or what - I've never removed this piece before. Just take your time and gently feel your way through it until they release.
Once you get that done you should be able to access the mouth that holds the speaker in place. Just remove the screws. I think there are 4 of them.
You can not go at it from the trunk because the piece is originally installed from inside the cabin, facing the rear. If you go at it from the trunk you won't be able to access the screws that hold the speaker in place.
Like I said, I just went into the garage and looked at my rear seat and middle panel, and based on my doing the same thing that I just told you to do, this is how I would go about getting the cover off.
Here is a link to a parts diagram of the rear speaker -- might help to see it.
http://tinyurl.com/24gpkw (look at the left side of the diagram for the center speaker...and get a load of the price that Honda wants for that speaker ......LMAO! $145.43 !!!!!!! But you get a discount at Majestic, $108.00......) It's just a 6x9 but I think it might have some kind of pre-amp on it or something, if I remember correctly)
Also, I have uploaded a 4th gen manual (doesn't cover all aspects of the OBDII 1996 model, but it will give you the basics for most things 4th gen.
Here's the manual download link:
http://www.mediafire.com/?1njqxn1dmtg
You can go to the "body" section of the manual and look at the rear seat removal section and the console removal section and get an idea of how everything is put together back there.
Hope this was helpful. Come back and let us know if you have success.