How to transfer from tape to digital

Great subject & I'm sure you'll get heaps of tips.
Sounds like you are off to a good start so far, so here's how I would go.
There is a free software called Audacity which is pretty easy to use, has a whole load of good help boards on the web & will also let you edit audio to remove things like clicks, add EQ & effects, speed up or slow down things &, most importantly record in just about any format you wish.
It'll run on both Windows & OSX.
Your source machine sounds right too. Some like to run via an amp, but I always maintain that adding components into the chain just adds to the distortion & colouration.
If you get Audacity installed play about with it until you get results you're happy with.
One thing with tapes that I always do is to fully fast forward & then rewind them before I start any transfer. It evens out the tension on tapes that may have been sitting idle for a while.
Next up, use decent cables if possible. Cables are always the weakest link in any hifi & tend to be the first thing to start failing.
Whatever you do, don't transfer at too high a volume. Have your volume cranked up full, start running the tape with Audacity on pause & change the recording levels so that you don't hit red lights on the sound meter. If you see the reds lighting up, your recording will "clip" & suffer from distortion.
When you're happy with the levels, get the tape started again, unpause Audacity & leave it to finish the side of the tape.
When the first side is done, pause Audacity (don't stop it). Flip the tape, start it up & unpause.
Don't worry about gaps in the music from the leader tape, as you can cut them out when you've done transferring.
At the end of side 2, then stop Audacity & you should have a single, huge audio file in the format of your choice.
The beauty of the one file is that if you do adjust the EQ of reduce hiss or however you edit, you only need to do it the once & the whole show will then sound more uniform before you split it into tracks.
My suggestion to you would be to set up Audacity to record in as high a definition as possible to allow you to get the best out of the audio for any editing work you might want to do. That would be 24bit, 48kHz flac. You could go nuts & go 24bit, 192kHz, but it's not usually necessary.
If you're pushed for disc space or not bothered about hi res audio so much, then 16bit, 44.1kHz is CD quality audio resolution & what collectors typically look for.
You could even save as an MP3, but if you have the tool to deliver the highest possible quality, then there isn't much point & anyone you share with can simply convert it if they wish for their own media players. The point being, that a flac (or even wav) file set offers people both options (lossless & lossy)
Play about for a bit with a handful of tracks off a random tape until you're comfortable & then let's start getting onto editing.
Of course, no harm in sharing the huge single files either. Pretty sure you'll get a lot of help to edit if you ask

If you're using the RCA outs, then you're relying on the Mac to amplify it, whereas a direct connection from the headphone out to the Mac will already be amplified (but the voltage shouldn't be anything to worry about), so either works. It's just a question of experimenting until you're happy you have the best result.
I am taking the headphone jack out of a Denon tape deck into my Mac.
I also use Amadeus Pro for the transfer, but keep Audacity on hand for other editing, as splitting tracks is a bit easier with Audacity I find.

It should be really. The only thing you might want to check is the distortion level difference between the RCA outs & the headphone out. Some headphone amps can be a little cr@ppy on that front.
What tape deck are you using just out of interest?

Oh, you'll be fine with that.

I would chose RCA outputs... audio is more "clean" than miniplug from front deck: this one use volume level from deck player. It would be necesary if you need to digitalize some vinyl from a (old) turntable, but not from tapes. RCA outputs are more "reliable"...
I hope my english is good enough..