New to the sport

That depends on what you call "shooting". There are so many different facets to the shooting sports and air guns have equivalents to all of them! Everything from backyard plinking, target shooting, and hunting to ISSF 10 meter and extreme benchrest.
I'll assume that you're talking about pellet rifles and not BB guns. Also, that you're not quite ready for the expense of PCP guns and all of their associated accoutrements. That leaves CO2 and piston powered guns, as they will be more-or-less self contained to allow you to just grab the gun and pellets while heading out the door.
Since you said it's for just starting out, I think we can eliminate high-end guns and competitions, too. Realistically, any gun can be used for shooting cans and bottle caps. What really matters in every type of shooting is accuracy. As Wyatt Earp famously said, "Speed is fine, but accuracy is final."
With all of that in mind, let's continue. I don't know what your budget is for this endeavor. So, I'll just give you a couple of ideas that aren't too very expensive but should be good starter guns for you:

$60 Crosman M4-177 - this is a pump-up rifle and accurate to about 10 yards. I shoot backyard mice with it from my kitchen.
$130 Krale Artemis CP2 - a fun little CO2 pellet gun in .177 or .22 with about 12 FPE (ft/lbs) good for close rattting.
$190 Crosman Mag-fire Mission - a multi-shot break barrel in .177 or .22 with about 14 ft/lbs of energy for rats/pigeons.
$270 Benjamin Trail Nitro Piston 2 Elite - a durable and field-worthy companion for hunting with 16 FPE.

Others will have their favorites, such as Beeman, RWS, Diana, Hatsan, etc. and you can't go wrong with those brands.
Let us all know what you decide to get, how you're using it, and (after some time shooting it) whether it's "The One" for you. We have all been bitten pretty hard by this bug and are curious about what might be the next pony in our corral.
Good luck,
~ Mondo

If you are not interested in hunting but are happy to go with back yard plinking at cans, balloons, wind-fall fruit, rotating/swinging/exploding targets made of wood, metal, plaster of Paris, paper, cardboard, plastic or whatever else you can think of come to that, then you would be much like me and in that case, these are some of the air rifles I have and would recommend and why:

Firstly, they are all 12g CO2 cartridge/capsule operated which I find convenient, easy to load and cheap to buy especially if bought in larger quantities.

All these air rifles are pretty accurate plinking wise, so I won’t go into detail for each.

Crosman 1077 .177 pellet - one of the cheapest rifles, light in weight, 12 shot magazine, spare magazines cheap to buy. Stick with the Black stock … why pay around £50 more for the plastic wood effect? … I can’t see the point.

Umarex Ruger 10/22 .177 pellet a replica air rifle – about twice the price of the Crosman 1077, it is heavier, sits snug in the shoulder, has a better feel to it, smoother, more solid build and action simply better all round. Mine has a Hawke Vantage 3-9 x 40 AO Mil Dot scope so of course the accuracy goes up. But if you don’t want a scope, by cocking the charging handle you can turn it from double action into single action reducing the trigger pull from 10lbs down to 3lbs thus enabling better accuracy.

SMK CR600W Single/Multi-shot .177 pellet – this air rifle has a wood stock with a lovely feel to it, sitting snug into the shoulder and against the cheek, it is fairly light and a dream to use the bolt action loading the pellets either singly in the tray, or by using the multi-shot magazines. These are not cheap to by and take a little getting used to, but I now have the hang of it and have several spares, as I just love the action of this rifle and it’s it a pure joy to use. I do have a scope on top though, just to spice things up a little.

Umarex Legends Cowboy - a replica in the style of the 1894 Winchester Model … a must for pure fun value. Load the 10 shells into the side of the rifle with each shell containing a BB (I bought the pellet shells myself and therefore loaded the back of each shell with a .177 pellet and it works just fine). Pull down the under-lever action feature which loads the shell, fire, pull the lever again and the spent shell pops out to be replaced with a new one – marvellous! – just buy loads of shells and to heck with the cost!

Air Venturi Hellboy M4 Hellraiser .177 BB – this is the last one I am recommending and again this is for fun factor value. It is a carbine with a 18 round BB magazine which also takes the CO2 cartridge. The rear sights can be removed to expose the Picatinny rail for adding Red-dot etc. and the rear stock is adjustable using a button. The spare magazines are not cheap because they are mostly metal and dual purpose, but are a must as you go through the BBs before you know it and you just gotta keep on replacing them like in the movies!

Okey-dokey, hope that gives you some things to think about on the lower power, back garden, plinking type of air gun fun.