Yup, that'll do it. The LM317T, being technically a 1.25v voltage regulator, will provide 1.25v higher than the adjustment pin, which will generate 18.38mA using a 68ohm resistor. This configuration is what makes it constant current. If you want a little more current, a 56 ohm will generate 22.3mA. The three LEDs will drop around 10.5v, add the "overhead" voltage of 1.25v for the 317T, and a little extra for stable running, will be a good match for a 4s pack with a 3.2v/cell cutoff. And the good news is that no changes are needed if you decide to run 5s, 6s, or even more.
Either way, the resistor needs to be at least 1/8w or more, which should be easy to find. BTW: those 68 and 56 values are common values, but RS doesn't seem to have a 56ohm on their site. You can either use a 68 ohm or use 150ohm+100ohm resistors in parallel to give you 60ohm total.
If you want a total of 6 LEDs, you could probably get away making a "3s2p" LED pack (

) and use a single 317T IC, but due to LED characteristic differences, I would use a 317T, resistor, and 3 LEDs for each side.
So, parts list for two circuits:
2 x LM317T
2 x 68ohm resistor (or 2x100ohm and 2x150ohm) 1/8w
6 x LEDs
Just be sure you hook up the 317T pins correctly or it won't work. There should be a device diagram on the package, but you can find the pinout at all kinds of places on the web.