[Proposal] Add MLP transcoders

[dtch1997]

Proposal

Support training, loading, and inference of MLP transcoders.

Motivation

MLP transcoders were trained by Jacob Dunefsky and Philippe Chlenski and have been shown to be useful.

Pitch

• Implement a HookedTranscoder class analogous to HookedSAE and using similar functionality.
• Implement a transcoder training runner.
• Support loading pre-trained transcoder checkpoints.

Checklist

• I have checked that there is no similar issue in the repo (required)

[dtch1997]

I'm likely going to be rather hacky with the first pass of the implementation, and possibly duplicate a bunch of code in order to have maximum freedom to make changes. Refactoring can be done later.

[anthonyduong9]

@dtch1997 are you planning on requesting a review on #183? If not, I'd like to work on this issue/build on your PR!

[OmarMohammed88]

any updates regarding adding it ?

[anthonyduong9]

@OmarMohammed88 yeah, I chatted in person with @dtch1997 about this, and I plan to complete it, but after #425 is merged. I'll assign myself to the issue.

[shivam-raval96]

@anthonyduong9 Hi, my friend and I are also considering working in this direction. Would you be available to meet and chat in the coming days? Our emails are sraval@g.harvard.edu and anton_poletaev@outlook.com. Looking forward to chatting:)

[anthonyduong9]

@shivam-raval96 hi, interesting! I'd be available. Could you book a meeting at https://calendly.com/anthonyduong9?