That Budget was the start of several years of political misfortune for the Keating government, and Swan was one of many parliamentarians to pay the price with the loss of their seats in the 1996 poll. One term was unlikely to satiate his passion and ambition, however, and he worked hard to claw back ground in Lilley. In the meantime, he joined the staff of opposition leader Kim Beazley, further cementing their established friendship. Swan would vote and organise for Beazley in every leadership ballot.
In the 1998 election, a national swing to Labor and Swan’s well-honed stump campaigning skills saw him re-elected as the member for Lilley in a clear result. His previous political experience saw him appointed to the shadow front bench, where Beazley gave him the family and community services portfolio. When Labor lost the 2001 election, Beazley was replaced as leader by Simon Crean, but Swan kept his portfolio and also became manager of opposition business in the House of Representatives, taking responsibility for Labor’s tactics in the House.
Crean’s leadership tenure was troubled, and poor poll results prompted speculation that Beazley would return to the position. Swan was a sharp internal critic of Crean’s performance, as were Stephen Smith, Anthony Albanese and Stephen Conroy. Outspoken Crean supporter Mark Latham labelled Swan and the other detractors ‘roosters’, a term that the quartet took as a compliment.
In June 2003, Beazley announced a challenge to Crean’s leadership, and Swan, Smith and Conroy publicly supported the return of their friend. There was not enough support for change, and Crean won the vote by fifty-eight votes to thirty-four. But few thought that this was the end of the matter. Indeed, Beazley refused to rule out another challenge. While Smith resigned from the shadow Cabinet as a matter of principle due to his support of the challenge, Crean and Swan agreed that resignation from the role of opposition business manager would be enough in Swan’s case. Latham, whom Crean was grooming for promotion, was appointed in Swan’s place.
In November 2003, a group of senior front benchers told Crean that they believed he had lost the support of the Caucus following continued poor polling. Swan and his colleagues were confident that Beazley could defeat Crean, but events were to take a different course. Counselled by his supporters, Crean decided not to contest the leadership and instead threw his support behind Latham, whom he had earlier made shadow treasurer. There was enough of a desire for generational change to see Latham elected leader by forty-seven votes to Beazley’s forty-five. The new Labor leader invited Stephen Smith to return to the shadow Cabinet and retained Swan as shadow minister for families.
Latham’s initial popularity filled the Labor Party with hope that its long run in opposition might be coming to an end. But attacks on Latham’s lack of experience and meagre economic credentials took their toll. While Latham had performed well to begin with, not enough had been done to build Labor’s economic credibility, and the Howard government ran an effective scare campaign about likely interest rate rises under a Latham administration. Despite opinion polls showing a very close result in the 2004 federal election, Labor suffered a 2 per cent swing against it and the net loss of five seats—a devastating result when victory had been expected.
On assuming the leadership, Latham had acceded to Crean’s request to be shadow treasurer. Following the election defeat, Crean knew that this position was no longer sustainable and transferred portfolios. Latham has said he was lobbied by Julia Gillard, Lindsay Tanner and Kevin Rudd for the shadow treasurership. However, he settled on Swan. He later claimed that he had wanted to damage Swan politically because he knew he wasn’t up to the job. If this is true, it was an extraordinarily self-damaging act of stupidity for a leader. A more accurate reason is probably the one found in Latham’s published diaries: ‘In the end, I have gone with the Queenslander. It might help us up there—he works hard and gets around the media.’2 Latham told Swan at the time that he had chosen him because of his performance in holding the government to account in the 2004 election campaign.
Swan had worked on a book during his time as shadow families minister and it was published soon after his ascension to the shadow Treasury portfolio. Titled Postcode: the Splintering of a Nation, it was a serious work containing detailed research, and an impassioned plea for a focus on equality. Swan gave an insight into the approach he would take as treasurer by writing:
In emphasising sound economic management we must never follow our opponents into the trap of turning politics into a mere exercise in accountancy. Budgets are also moral documents revealing our priorities … We must never lose sight of moral as well as economic arguments. For me, social justice and inclusion are moral issues. And the most significant question in politics is what we use our prosperity for.3
Latham did not last long after his 2004 election defeat, resigning in January 2005. The Labor Party toyed with a move to Rudd or Gillard as leader, but Swan’s support for Beazley’s return was clear and unwavering. A majority of the Caucus agreed, and Beazley was re-elected unopposed.
Swan’s most testing time as shadow treasurer coincided with the 2005 Budget. Awash with increased revenues due to the mining boom, treasurer Peter Costello was able to announce more than $36 billion in tax cuts. It was higher-income earners who were to benefit most substantially, with the top tax rates of 47 per cent and 45 per cent both to be cut by 2 per cent. Beazley and Swan agreed on budget night to oppose the tax cuts on the basis of fairness, and to insist that Labor
