Advanced Text Mining with R and quanteda
Cornelius Puschmann, @cbpuschmann
May 16th, 2018
MZES, University of Mannheim
Overview
- Why do text mining/automated content analysis?
- Why use quanteda?
- Getting data into quanteda with readtext
- Compiling and working with corpora
- Measures and metrics for words and texts
- Applying lexicons
- Topic modeling
- Automating content analysis
Download the data and lexicons here.
Compiling and working with corpora
Before getting started, we install and load the required libraries.
if(!require("readtext")) install.packages("readtext")
if(!require("quanteda")) install.packages("quanteda")
if(!require("lubridate")) install.packages("lubridate")
if(!require("tidyverse")) install.packages("tidyverse")
library("readtext")
library("quanteda")
library("lubridate")
library("tidyverse")
theme_set(theme_bw())We start our analysis by reading in the Harry Potter corpus and assigning basic meta data. We use the readtext() function to load the data and the corpus() function to generate quanteda corpus from it.
# Einlesen der Harry Potter-Textdateien mit readtext() und Erstellung eines quanteda-Korpus nebst von Metadaten mit corpus()
quelldateien <- dir("data/harrypotter", pattern = "chapter_[0-9]{2}.txt")
daten.harrypotter <- readtext(paste("data/harrypotter", quelldateien, sep = "/"))
daten.harrypotter$doc_id <- sprintf("%02d", 1:38) # praktischere Benennung der Kapitel
korpus <- corpus(daten.harrypotter, docid_field = "doc_id")
korpus.stats <- summary(korpus, n = 1000000)Next we generate a set of basic corpus statistics with the functions ndoc(), ntoken(), ntyope() and nsentence().
korpusCorpus consisting of 38 documents and 0 docvars.ndoc(korpus) # number of documents[1] 38ntoken(korpus) # number of tokens 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17
7239 8634 5952 7385 7095 8625 5806 5458 10562 7496 7401 10761 11111 10344 8816 7677 8615
18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
8985 8439 8385 9760 10400 10240 9789 10184 10866 9699 10298 9547 10802 9891 8317 4970 6557
35 36 37 38
9868 4943 10139 9931 ntype(korpus) # number of types 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20
1780 1782 1506 1663 1553 1938 1512 1279 2023 1783 1783 2209 2123 2056 1832 1568 1865 1878 1865 1678
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38
1967 2102 2124 1994 2192 2140 1986 2142 1916 2149 2186 1671 1237 1439 1759 1187 1873 2050 nsentence(korpus) # number of sentences 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
408 575 367 440 426 410 289 303 630 440 373 620 617 539 457 376 500 509 377 538 542 530 534 565 564
26 27 28 29 30 31 32 33 34 35 36 37 38
575 565 615 516 601 487 519 283 379 537 271 540 560 korpus.statsCorpus consisting of 38 documents:
Text Types Tokens Sentences
01 1780 7239 408
02 1782 8634 575
03 1506 5952 367
04 1663 7385 440
05 1553 7095 426
06 1938 8625 410
07 1512 5806 289
08 1279 5458 303
09 2023 10562 630
10 1783 7496 440
11 1783 7401 373
12 2209 10761 620
13 2123 11111 617
14 2056 10344 539
15 1832 8816 457
16 1568 7677 376
17 1865 8615 500
18 1878 8985 509
19 1865 8439 377
20 1678 8385 538
21 1967 9760 542
22 2102 10400 530
23 2124 10240 534
24 1994 9789 565
25 2192 10184 564
26 2140 10866 575
27 1986 9699 565
28 2142 10298 615
29 1916 9547 516
30 2149 10802 601
31 2186 9891 487
32 1671 8317 519
33 1237 4970 283
34 1439 6557 379
35 1759 9868 537
36 1187 4943 271
37 1873 10139 540
38 2050 9931 560
Source: /Users/cp/Dropbox/Lehre/Advanced Text Mining with quanteda/* on x86_64 by cp
Created: Wed May 9 15:19:40 2018
Notes: Let’s examine (a) the number of tokens per chapter, (b) the number of types per chapter and (c) the number of sentences per chapter.
# Plotten von (a) Tokens pro Kapitel (b) Typen pro Kapitel (c) Sätze pro Kapitel
ggplot(korpus.stats, aes(Text, Tokens, group=1)) + geom_line() + geom_point() + ggtitle("Tokens pro Kapitel")
ggplot(korpus.stats, aes(Text, Types, group=1)) + geom_line() + geom_point() + ggtitle("Types pro Kapitel")
ggplot(korpus.stats, aes(Text, Sentences, group=1)) + geom_line() + geom_point() + ggtitle("Sätze pro Kapitel")
# ...und zusammen
ggplot(korpus.stats %>% gather(Types, Tokens, Sentences, key = "Einheit", value = "Anzahl"), aes(Text, Anzahl, group = Einheit, col = Einheit)) + geom_line(size = 1) + ggtitle("Tokens, Types und Sätze pro Kapitel")
Plotteing the type-token ratio (TTR)
ggplot(korpus.stats, aes(Tokens, Types, group=1, label = Text)) + geom_smooth(method = "lm", se = FALSE) + geom_text(check_overlap = T) + ggtitle("Type-Token ratio per chapter")
# Korpora filtern, umformen, und mit Metadaten versehen
# Inhalt des ersten Kapitels wiedergeben
korpus[1] 01
"CHAPTER ONE DUDLEY DEMENTED he hottest day of the summer so far was drawing to a close and a drowsy silence lay over the large, square houses of Privet Drive. Cars that were usually gleaming stood dusty in their drives and lawns that were once emerald green lay parched and yellowing; the use of hosepipes had been banned due to drought. Deprived of their usual car-washing and lawn-mowing pursuits, the inhabitants of Privet Drive had retreated into the shade of their cool houses, windows thrown wide in the hope of tempting in a nonexistent breeze. The only person left outdoors was a teenage boy who was lying flat on his back in a flower bed outside number four. He was a skinny, black-haired, bespectacled boy who had the pinched, slightly unhealthy look of someone who has grown a lot in a short space of time. His jeans were torn and dirty, his T-shirt baggy and faded, and the soles of his trainers were peeling away from the uppers. Harry Potter's appearance did not endear him to the neighbors, who were the sort of people who thought scruffiness ought to be punishable by law, but as he had hidden himself behind a large hydrangea bush this evening he was quite invisible to passersby. In fact, the only T 1 CHAPTER ONE way he would be spotted was if his Uncle Vernon or Aunt Petunia stuck their heads out of the living room window and looked straight down into the flower bed below. On the whole, Harry thought he was to be congratulated on his idea of hiding here. He was not, perhaps, very comfortable lying on the hot, hard earth, but on the other hand, nobody was glaring at him, grinding their teeth so loudly that he could not hear the news, or shooting nasty questions at him, as had happened every time he had tried sitting down in the living room and watching television with his aunt and uncle. Almost as though this thought had fluttered through the open window, Vernon Dursley, Harry's uncle, suddenly spoke. \"Glad to see the boy's stopped trying to butt in. Where is he anyway?\" \"I don't know,\" said Aunt Petunia unconcernedly. \"Not in the house.\" Uncle Vernon grunted. \"Watching the news . . .\" he said scathingly. \"I'd like to know what he's really up to. As if a normal boy cares what's on the news -- Dudley hasn't got a clue what's going on, doubt he knows who the Prime Minister is! Anyway, it's not as if there'd be anything about his lot on our news --\" \"Vernon, shh !\" said Aunt Petunia. \"The window's open!\" \"Oh -- yes -- sorry, dear . . .\" The Dursleys fell silent. Harry listened to a jingle about Fruit 'N Bran breakfast cereal while he watched Mrs. Figg, a batty, cat-loving old lady from nearby Wisteria Walk, amble slowly past. She was frowning and muttering to herself. Harry was very pleased that he was concealed behind the bush; Mrs. Figg had recently taken to asking him around for tea whenever she met him in the street. She had rounded the corner and vanished from view before Uncle Vernon's voice floated out of the window again. 2 DUDLEY DEMENTED \"Dudders out for tea?\" \"At the Polkisses',\" said Aunt Petunia fondly. \"He's got so many little friends, he's so popular . . .\" Harry repressed a snort with difficulty. The Dursleys really were astonishingly stupid about their son, Dudley; they had swallowed all his dim-witted lies about having tea with a different member of his gang every night of the summer holidays. Harry knew perfectly well that Dudley had not been to tea anywhere; he and his gang spent every evening vandalizing the play park, smoking on street corners, and throwing stones at passing cars and children. Harry had seen them at it during his evening walks around Little Whinging; he had spent most of the holidays wandering the streets, scavenging newspapers from bins along the way. The opening notes of the music that heralded the seven o'clock news reached Harry's ears and his stomach turned over. Perhaps tonight -- after a month of waiting -- would be the night -- \"Record numbers of stranded holidaymakers fill airports as the Spanish baggage-handlers' strike reaches its second week --\" \"Give 'em a lifelong siesta, I would,\" snarled Uncle Vernon over the end of the newsreader's sentence, but no matter: Outside in the flower bed, Harry's stomach seemed to unclench. If anything had happened, it would surely have been the first item on the news; death and destruction were more important than stranded holidaymakers. . . . He let out a long, slow breath and stared up at the brilliant blue sky. Every day this summer had been the same: the tension, the expectation, the temporary relief, and then mounting tension again . . . and always, growing more insistent all the time, the question of why nothing had happened yet. . . . He kept listening, just in case there was some small clue, not recognized for what it really was by the Muggles -- an unexplained disappearance, perhaps, or some strange accident . . . but the 3 CHAPTER ONE baggage-handlers' strike was followed by news on the drought in the Southeast (\"I hope he's listening next door!\" bellowed Uncle Vernon, \"with his sprinklers on at three in the morning!\"); then a helicopter that had almost crashed in a field in Surrey, then a famous actress's divorce from her famous husband (\"as if we're interested in their sordid affairs,\" sniffed Aunt Petunia, who had followed the case obsessively in every magazine she could lay her bony hands on). Harry closed his eyes against the now blazing evening sky as the newsreader said, \"And finally, Bungy the budgie has found a novel way of keeping cool this summer. Bungy, who lives at the Five Feathers in Barnsley, has learned to water-ski! Mary Dorkins went to find out more. . . .\" Harry opened his eyes again. If they had reached water-skiing budgerigars, there was nothing else worth hearing. He rolled cautiously onto his front and raised himself onto his knees and elbows, preparing to crawl out from under the window. He had moved about two inches when several things happened in very quick succession. A loud, echoing crack broke the sleepy silence like a gunshot; a cat streaked out from under a parked car and flew out of sight; a shriek, a bellowed oath, and the sound of breaking china came from the Dursleys' living room, and as though Harry had been waiting for this signal, he jumped to his feet, at the same time pulling from the waistband of his jeans a thin wooden wand as if he were unsheathing a sword. But before he could draw himself up to full height, the top of his head collided with the Dursleys' open window, and the resultant crash made Aunt Petunia scream even louder. Harry felt as if his head had been split in two; eyes streaming, he swayed, trying to focus on the street and spot the source of the noise, but he had barely staggered upright again when two large purple hands reached through the open window and closed tightly around his throat. 4 DUDLEY DEMENTED \"Put -- it -- away !\" Uncle Vernon snarled into Harry's ear. \"Now ! Before -- anyone -- sees !\" \"Get -- off -- me!\" Harry gasped; for a few seconds they struggled, Harry pulling at his uncle's sausage-like fingers with his left hand, his right maintaining a firm grip on his raised wand. Then, as the pain in the top of Harry's head gave a particularly nasty throb, Uncle Vernon yelped and released Harry as though he had received an electric shock -- some invisible force seemed to have surged through his nephew, making him impossible to hold. Panting, Harry fell forward over the hydrangea bush, straightened up, and stared around. There was no sign of what had caused the loud cracking noise, but there were several faces peering through various nearby windows. Harry stuffed his wand hastily back into his jeans and tried to look innocent. \"Lovely evening!\" shouted Uncle Vernon, waving at Mrs. Number Seven, who was glaring from behind her net curtains. \"Did you hear that car backfire just now? Gave Petunia and me quite a turn!\" He continued to grin in a horrible, manic way until all the curious neighbors had disappeared from their various windows, then the grin became a grimace of rage as he beckoned Harry back toward him. Harry moved a few steps closer, taking care to stop just short of the point at which Uncle Vernon's outstretched hands could resume their strangling. \"What the devil do you mean by it, boy?\" asked Uncle Vernon in a croaky voice that trembled with fury. \"What do I mean by what?\" said Harry coldly. He kept looking left and right up the street, still hoping to see the person who had made the cracking noise. \"Making a racket like a starting pistol right outside our --\" \"I didn't make that noise,\" said Harry firmly. Aunt Petunia's thin, horsey face now appeared beside Uncle Vernon's wide, purple one. She looked livid. 5 CHAPTER ONE \"Why were you lurking under our window?\" \"Yes -- yes, good point, Petunia! What were you doing under our window, boy ?\" \"Listening to the news,\" said Harry in a resigned voice. His aunt and uncle exchanged looks of outrage. \"Listening to the news! Again ?\" \"Well, it changes every day, you see,\" said Harry. \"Don't you be clever with me, boy! I want to know what you're really up to -- and don't give me any more of this listening to the news tosh! You know perfectly well that your lot . . .\" \"Careful, Vernon!\" breathed Aunt Petunia, and Uncle Vernon lowered his voice so that Harry could barely hear him, \". . . that your lot don't get on our news!\" \"That's all you know,\" said Harry. The Dursleys goggled at him for a few seconds, then Aunt Petunia said, \"You're a nasty little liar. What are all those --\" she too lowered her voice so that Harry had to lip-read the next word, \"-- owls -- doing if they're not bringing you news?\" \"Aha!\" said Uncle Vernon in a triumphant whisper. \"Get out of that one, boy! As if we didn't know you get all your news from those pestilential birds!\" Harry hesitated for a moment. It cost him something to tell the truth this time, even though his aunt and uncle could not possibly know how bad Harry felt at admitting it. \"The owls . . . aren't bringing me news,\" said Harry tonelessly. \"I don't believe it,\" said Aunt Petunia at once. \"No more do I,\" said Uncle Vernon forcefully. \"We know you're up to something funny,\" said Aunt Petunia. \"We're not stupid, you know,\" said Uncle Vernon. \"Well, that's news to me,\" said Harry, his temper rising, and before the Dursleys could call him back, he had wheeled about, crossed the 6 DUDLEY DEMENTED front lawn, stepped over the low garden wall, and was striding off up the street. He was in trouble now and he knew it. He would have to face his aunt and uncle later and pay the price for his rudeness, but he did not care very much just at the moment; he had much more pressing matters on his mind. Harry was sure that the cracking noise had been made by someone Apparating or Disapparating. It was exactly the sound Dobby the house-elf made when he vanished into thin air. Was it possible that Dobby was here in Privet Drive? Could Dobby be following him right at this very moment? As this thought occurred he wheeled around and stared back down Privet Drive, but it appeared to be completely deserted again and Harry was sure that Dobby did not know how to become invisible. . . . He walked on, hardly aware of the route he was taking, for he had pounded these streets so often lately that his feet carried him to his favorite haunts automatically. Every few steps he glanced back over his shoulder. Someone magical had been near him as he lay among Aunt Petunias dying begonias, he was sure of it. Why hadn't they spoken to him, why hadn't they made contact, why were they hiding now? And then, as his feeling of frustration peaked, his certainty leaked away. Perhaps it hadn't been a magical sound after all. Perhaps he was so desperate for the tiniest sign of contact from the world to which he belonged that he was simply overreacting to perfectly ordinary noises. Could he be sure it hadn't been the sound of something breaking inside a neighbor's house? Harry felt a dull, sinking sensation in his stomach and, before he knew it, the feeling of hopelessness that had plagued him all summer rolled over him once again. . . . Tomorrow morning he would be awoken by the alarm at five 7 CHAPTER ONE o'clock so that he could pay the owl that delivered the Daily Prophet -- but was there any point in continuing to take it? Harry merely glanced at the front page before throwing it aside these days; when the idiots who ran the paper finally realized that Voldemort was back it would be headline news, and that was the only kind Harry cared about. If he was lucky, there would also be owls carrying letters from his best friends, Ron and Hermione, though any expectation he had had that their letters would bring him news had long since been dashed. \"We can't say much about you-know-what, obviously. . . .\" \"We've been told not to say anything important in case our letters go astray. . . .\" \"We're quite busy but I can't give you details here. . . .\" \"There's a fair amount going on, we'll tell you everything when we see you. . . .\" But when were they going to see him? Nobody seemed too bothered with a precise date. Hermione had scribbled, \"I expect we'll be seeing you quite soon\" inside his birthday card, but how soon was soon? As far as Harry could tell from the vague hints in their letters, Hermione and Ron were in the same place, presumably at Ron's parents' house. He could hardly bear to think of the pair of them having fun at the Burrow when he was stuck in Privet Drive. In fact, he was so angry at them that he had thrown both their birthday presents of Honeydukes chocolates away unopened, though he had regretted this after eating the wilting salad Aunt Petunia had provided for dinner that night. And what were Ron and Hermione busy with? Why wasn't he, Harry, busy? Hadn't he proved himself capable of handling much more than they? Had they all forgotten what he had done? Hadn't it been he who had entered that graveyard and watched Cedric being murdered and been tied to that tombstone and nearly killed . . . ? Don't think about that, Harry told himself sternly for the hundredth time that summer. It was bad enough that he kept revisiting the graveyard in his nightmares, without dwelling on it in his waking moments too. 8 DUDLEY DEMENTED He turned a corner into Magnolia Crescent; halfway along he passed the narrow alleyway down the side of a garage where he had first clapped eyes on his godfather. Sirius, at least, seemed to understand how Harry was feeling; admittedly his letters were just as empty of proper news as Ron and Hermione's, but at least they contained words of caution and consolation instead of tantalizing hints: \"I know this must be frustrating for you. . . .\" \"Keep your nose clean and everything will be okay. . . .\" \"Be careful and don't do anything rash. . . .\" Well, thought Harry, as he crossed Magnolia Crescent, turned into Magnolia Road, and headed toward the darkening play park, he had (by and large) done as Sirius advised; he had at least resisted the temptation to tie his trunk to his broomstick and set off for the Burrow by himself. In fact Harry thought his behavior had been very good considering how frustrated and angry he felt at being stuck in Privet Drive this long, reduced to hiding in flower beds in the hope of hearing something that might point to what Lord Voldemort was doing. Nevertheless, it was quite galling to be told not to be rash by a man who had served twelve years in the wizard prison, Azkaban, escaped, attempted to commit the murder he had been convicted for in the first place, then gone on the run with a stolen hippogriff. . . . Harry vaulted over the locked park gate and set off across the parched grass. The park was as empty as the surrounding streets. When he reached the swings he sank onto the only one that Dudley and his friends had not yet managed to break, coiled one arm around the chain, and stared moodily at the ground. He would not be able to hide in the Dursleys' flower bed again. Tomorrow he would have to think of some fresh way of listening to the news. In the meantime, he had nothing to look forward to but another restless, disturbed night, because even when he escaped nightmares about Cedric he had unsettling dreams about long dark corridors, all finishing in dead ends and locked doors, which he supposed had something to do with the 9 CHAPTER ONE trapped feeling he had when he was awake. Often the old scar on his forehead prickled uncomfortably, but he did not fool himself that Ron or Hermione or Sirius would find that very interesting anymore. . . . In the past his scar hurting had warned that Voldemort was getting stronger again, but now that Voldemort was back they would probably remind him that its regular irritation was only to be expected. . . . Nothing to worry about . . . old news . . . The injustice of it all welled up inside him so that he wanted to yell with fury. If it hadn't been for him, nobody would even have known Voldemort was back! And his reward was to be stuck in Little Whinging for four solid weeks, completely cut off from the magical world, reduced to squatting among dying begonias so that he could hear about water-skiing budgerigars! How could Dumbledore have forgotten him so easily? Why had Ron and Hermione got together without inviting him along too? How much longer was he supposed to endure Sirius telling him to sit tight and be a good boy; or resist the temptation to write to the stupid Daily Prophet and point out that Voldemort had returned? These furious thoughts whirled around in Harry's head, and his insides writhed with anger as a sultry, velvety night fell around him, the air full of the smell of warm, dry grass and the only sound that of the low grumble of traffic on the road beyond the park railings. He did not know how long he had sat on the swing before the sound of voices interrupted his musings and he looked up. The streetlamps from the surrounding roads were casting a misty glow strong enough to silhouette a group of people making their way across the park. One of them was singing a loud, crude song. The others were laughing. A soft ticking noise came from several expensive racing bikes that they were wheeling along. Harry knew who those people were. The figure in front was unmistakably his cousin, Dudley Dursley, wending his way home, accompanied by his faithful gang. 10 DUDLEY DEMENTED Dudley was as vast as ever, but a year's hard dieting and the discovery of a new talent had wrought quite a change in his physique. As Uncle Vernon delightedly told anyone who would listen, Dudley had recently become the Junior Heavyweight Inter-School Boxing Champion of the Southeast. \"The noble sport,\" as Uncle Vernon called it, had made Dudley even more formidable than he had seemed to Harry in the primary school days when he had served as Dudley's first punching bag. Harry was not remotely afraid of his cousin anymore but he still didn't think that Dudley learning to punch harder and more accurately was cause for celebration. Neighborhood children all around were terrified of him -- even more terrified than they were of \"that Potter boy,\" who, they had been warned, was a hardened hooligan who attended St. Brutus's Secure Center for Incurably Criminal Boys. Harry watched the dark figures crossing the grass and wondered whom they had been beating up tonight. Look round, Harry found himself thinking as he watched them. Come on . . . look round . . . I'm sitting here all alone. . . . Come and have ago. . . . If Dudley's friends saw him sitting here, they would be sure to make a beeline for him, and what would Dudley do then? He wouldn't want to lose face in front of the gang, but he'd be terrified of provoking Harry. . . . It would be really fun to watch Dudley's dilemma; to taunt him, watch him, with him powerless to respond . . . and if any of the others tried hitting Harry, Harry was ready -- he had his wand . . . let them try . . . He'd love to vent some of his frustration on the boys who had once made his life hell -- But they did not turn around, they did not see him, they were almost at the railings. Harry mastered the impulse to call after them. . . . Seeking a fight was not a smart move. . . . He must not use magic. . . . He would be risking expulsion again. . . . Dudley's gang's voices died; they were out of sight, heading along Magnolia Road. 11 CHAPTER ONE There you go, Sirius, Harry thought dully. Nothing rash. Kept my nose clean. Exactly the opposite of what you'd have done . . . He got to his feet and stretched. Aunt Petunia and Uncle Vernon seemed to feel that whenever Dudley turned up was the right time to be home, and anytime after that was much too late. Uncle Vernon had threatened to lock Harry in the shed if he came home after Dudley again, so, stifling a yawn, still scowling, Harry set off toward the park gate. Magnolia Road, like Privet Drive, was full of large, square houses with perfectly manicured lawns, all owned by large, square owners who drove very clean cars similar to Uncle Vernon's. Harry preferred Little Whinging by night, when the curtained windows made patches of jewel-bright colors in the darkness and he ran no danger of hearing disapproving mutters about his \"delinquent\" appearance when he passed the householders. He walked quickly, so that halfway along Magnolia Road Dudley's gang came into view again; they were saying their farewells at the entrance to Magnolia Crescent. Harry stepped into the shadow of a large lilac tree and waited. \". . . squealed like a pig, didn't he?\" Malcolm was saying, to guffaws from the others. \"Nice right hook, Big D,\" said Piers. \"Same time tomorrow?\" said Dudley. \"Round at my place, my parents are out,\" said Gordon. \"See you then,\" said Dudley. \"Bye Dud!\" \"See ya, Big D!\" Harry waited for the rest of the gang to move on before setting off again. When their voices had faded once more he headed around the corner into Magnolia Crescent and by walking very quickly he soon came within hailing distance of Dudley, who was strolling along at his ease, humming tunelessly. \"Hey, Big D!\" 12 DUDLEY DEMENTED Dudley turned. \"Oh,\" he grunted. \"It's you.\" \"How long have you been `Big D' then?\" said Harry. \"Shut it,\" snarled Dudley, turning away again. \"Cool name,\" said Harry, grinning and falling into step beside his cousin. \"But you'll always be Ickle Diddykins to me.\" \"I said, SHUT IT!\" said Dudley, whose ham-like hands had curled into fists. \"Don't the boys know that's what your mum calls you?\" \"Shut your face.\" \"You don't tell her to shut her face. What about `popkin' and `Dinky Diddydums,' can I use them then?\" Dudley said nothing. The effort of keeping himself from hitting Harry seemed to be demanding all his self-control. \"So who've you been beating up tonight?\" Harry asked, his grin fading. \"Another ten-year-old? I know you did Mark Evans two nights ago --\" \"He was asking for it,\" snarled Dudley. \"Oh yeah?\" \"He cheeked me.\" \"Yeah? Did he say you look like a pig that's been taught to walk on its hind legs? 'Cause that's not cheek, Dud, that's true . . .\" A muscle was twitching in Dudley's jaw. It gave Harry enormous satisfaction to know how furious he was making Dudley; he felt as though he was siphoning off his own frustration into his cousin, the only outlet he had. They turned right down the narrow alleyway where Harry had first seen Sirius and which formed a shortcut between Magnolia Crescent and Wisteria Walk. It was empty and much darker than the streets it linked because there were no streetlamps. Their footsteps were muffled between garage walls on one side and a high fence on the other. 13 CHAPTER ONE \"Think you're a big man carrying that thing, don't you?\" Dudley said after a few seconds. \"What thing?\" \"That -- that thing you're hiding.\" Harry grinned again. \"Not as stupid as you look, are you, Dud? But I s'pose if you were, you wouldn't be able to walk and talk at the same time. . . .\" Harry pulled out his wand. He saw Dudley look sideways at it. \"You're not allowed,\" Dudley said at once. \"I know you're not. You'd get expelled from that freak school you go to.\" \"How d'you know they haven't changed the rules, Big D?\" \"They haven't,\" said Dudley, though he didn't sound completely convinced. Harry laughed softly. \"You haven't got the guts to take me on without that thing, have you?\" Dudley snarled. \"Whereas you just need four mates behind you before you can beat up a ten-year-old. You know that boxing title you keep banging on about? How old was your opponent? Seven? Eight?\" \"He was sixteen for your information,\" snarled Dudley, \"and he was out cold for twenty minutes after I'd finished with him and he was twice as heavy as you. You just wait till I tell Dad you had that thing out --\" \"Running to Daddy now, are you? Is his ickle boxing champ frightened of nasty Harry's wand?\" \"Not this brave at night, are you?\" sneered Dudley. \"This is night, Diddykins. That's what we call it when it goes all dark like this.\" \"I mean when you're in bed!\" Dudley snarled. He had stopped walking. Harry stopped too, staring at his cousin. From the little he could see of Dudley's large face, he was wearing a strangely triumphant look. \"What d'you mean, I'm not brave in bed?\" said Harry, completely 14 DUDLEY DEMENTED nonplussed. \"What -- am I supposed to be frightened of pillows or something?\" \"I heard you last night,\" said Dudley breathlessly. \"Talking in your sleep. Moaning.\" \"What d'you mean?\" Harry said again, but there was a cold, plunging sensation in his stomach. He had revisited the graveyard last night in his dreams. Dudley gave a harsh bark of laughter then adopted a high-pitched, whimpering voice. \" `Don't kill Cedric! Don't kill Cedric!' Who's Cedric -- your boyfriend?\" \"I -- you're lying --\" said Harry automatically. But his mouth had gone dry. He knew Dudley wasn't lying -- how else would he know about Cedric? \" `Dad! Help me, Dad! He's going to kill me, Dad! Boo-hoo!' \" \"Shut up,\" said Harry quietly. \"Shut up, Dudley, I'm warning you!\" \" `Come and help me, Dad! Mum, come and help me! He's killed Cedric! Dad, help me! He's going to --' Don't you point that thing at me !\" Dudley backed into the alley wall. Harry was pointing the wand directly at Dudley's heart. Harry could feel fourteen years' hatred of Dudley pounding in his veins -- what wouldn't he give to strike now, to jinx Dudley so thoroughly he'd have to crawl home like an insect, struck dumb, sprouting feelers -- \"Don't ever talk about that again,\" Harry snarled. \"D'you understand me?\" \"Point that thing somewhere else!\" \"I said, do you understand me ?\" \"Point it somewhere else !\" \"DO YOU UNDERSTAND ME?\" \"GET THAT THING AWAY FROM --\" Dudley gave an odd, shuddering gasp, as though he had been doused in icy water. 15 CHAPTER ONE Something had happened to the night. The star-strewn indigo sky was suddenly pitch-black and lightless -- the stars, the moon, the misty streetlamps at either end of the alley had vanished. The distant grumble of cars and the whisper of trees had gone. The balmy evening was suddenly piercingly, bitingly cold. They were surrounded by total, impenetrable, silent darkness, as though some giant hand had dropped a thick, icy mantle over the entire alleyway, blinding them. For a split second Harry thought he had done magic without meaning to, despite the fact that he'd been resisting as hard as he could -- then his reason caught up with his senses -- he didn't have the power to turn off the stars. He turned his head this way and that, trying to see something, but the darkness pressed on his eyes like a weightless veil. Dudley's terrified voice broke in Harry's ear. \"W-what are you d-doing? St-stop it!\" \"I'm not doing anything! Shut up and don't move!\" \"I c-can't see! I've g-gone blind! I --\" \"I said shut up!\" Harry stood stock-still, turning his sightless eyes left and right. The cold was so intense that he was shivering all over; goose bumps had erupted up his arms, and the hairs on the back of his neck were standing up -- he opened his eyes to their fullest extent, staring blankly around, unseeing . . . It was impossible. . . . They couldn't be here. . . . Not in Little Whinging . . . He strained his ears. . . . He would hear them before he saw them. . . . \"I'll t-tell Dad!\" Dudley whimpered. \"W-where are you? What are you d-do -- ?\" \"Will you shut up?\" Harry hissed, \"I'm trying to lis --\" But he fell silent. He had heard just the thing he had been dreading. There was something in the alleyway apart from themselves, some 16 DUDLEY DEMENTED thing that was drawing long, hoarse, rattling breaths. Harry felt a horrible jolt of dread as he stood trembling in the freezing air. \"C-cut it out! Stop doing it! I'll h-hit you, I swear I will!\" \"Dudley, shut --\" WHAM ! A fist made contact with the side of Harry's head, lifting Harry off his feet. Small white lights popped in front of Harry's eyes; for the second time in an hour he felt as though his head had been cleaved in two; next moment he had landed hard on the ground, and his wand had flown out of his hand. \"You moron, Dudley!\" Harry yelled, his eyes watering with pain, as he scrambled to his hands and knees, now feeling around frantically in the blackness. He heard Dudley blundering away, hitting the alley fence, stumbling. \"DUDLEY, COME BACK! YOU'RE RUNNING RIGHT AT IT!\" There was a horrible squealing yell, and Dudley's footsteps stopped. At the same moment, Harry felt a creeping chill behind him that could mean only one thing. There was more than one. \"DUDLEY, KEEP YOUR MOUTH SHUT! WHATEVER YOU DO, KEEP YOUR MOUTH SHUT! Wand!\" Harry muttered frantically, his hands flying over the ground like spiders. \"Where's -- wand -- come on -- Lumos !\" He said the spell automatically, desperate for light to help him in his search -- and to his disbelieving relief, light flared inches from his right hand -- the wand tip had ignited. Harry snatched it up, scrambled to his feet, and turned around. His stomach turned over. A towering, hooded figure was gliding smoothly toward him, hovering over the ground, no feet or face visible beneath its robes, sucking on the night as it came. Stumbling backward, Harry raised his wand. 17 CHAPTER ONE \"Expecto Patronum !\" A silvery wisp of vapor shot from the tip of the wand and the dementor slowed, but the spell hadn't worked properly; tripping over his feet, Harry retreated farther as the dementor bore down upon him, panic fogging his brain -- concentrate -- A pair of gray, slimy, scabbed hands slid from inside the dementor's robes, reaching for him. A rushing noise filled Harry's ears. \"Expecto Patronum !\" His voice sounded dim and distant. . . . Another wisp of silver smoke, feebler than the last, drifted from the wand -- he couldn't do it anymore, he couldn't work the spell -- There was laughter inside his own head, shrill, high-pitched laughter. . . . He could smell the dementor's putrid, death-cold breath, filling his own lungs, drowning him -- Think . . . something happy. . . . But there was no happiness in him. . . . The dementor's icy fingers were closing on his throat -- the high-pitched laughter was growing louder and louder, and a voice spoke inside his head -- \"Bow to death, Harry. . . . It might even be painless. . . . I would not know. . . . I have never died. . . .\" He was never going to see Ron and Hermione again -- And their faces burst clearly into his mind as he fought for breath -- \"EXPECTO PATRONUM !\" An enormous silver stag erupted from the tip of Harry's wand; its antlers caught the dementor in the place where the heart should have been; it was thrown backward, weightless as darkness, and as the stag charged, the dementor swooped away, batlike and defeated. \"THIS WAY!\" Harry shouted at the stag. Wheeling around, he sprinted down the alleyway, holding the lit wand aloft. \"DUDLEY? DUDLEY!\" He had run barely a dozen steps when he reached them: Dudley was curled on the ground, his arms clamped over his face; a second de 18 DUDLEY DEMENTED mentor was crouching low over him, gripping his wrists in its slimy hands, prizing them slowly, almost lovingly apart, lowering its hooded head toward Dudley's face as though about to kiss him. . . . \"GET IT!\" Harry bellowed, and with a rushing, roaring sound, the silver stag he had conjured came galloping back past him. The dementor's eyeless face was barely an inch from Dudley's when the silver antlers caught it; the thing was thrown up into the air and, like its fellow, it soared away and was absorbed into the darkness. The stag cantered to the end of the alleyway and dissolved into silver mist. Moon, stars, and streetlamps burst back into life. A warm breeze swept the alleyway. Trees rustled in neighboring gardens and the mundane rumble of cars in Magnolia Crescent filled the air again. Harry stood quite still, all his senses vibrating, taking in the abrupt return to normality. After a moment he became aware that his T-shirt was sticking to him; he was drenched in sweat. He could not believe what had just happened. Dementors here, in Little Whinging . . . Dudley lay curled up on the ground, whimpering and shaking. Harry bent down to see whether he was in a fit state to stand up, but then heard loud, running footsteps behind him; instinctively raising his wand again, he spun on his heel to face the newcomer. Mrs. Figg, their batty old neighbor, came panting into sight. Her grizzled gray hair was escaping from its hairnet, a clanking string shopping bag was swinging from her wrist, and her feet were halfway out of her tartan carpet slippers. Harry made to stow his wand hurriedly out of sight, but -- \"Don't put it away, idiot boy!\" she shrieked. \"What if there are more of them around? Oh, I'm going to kill Mundungus Fletcher!\" 19" # Korpus mit corpus_reshape umformen, so dass jeder Satz ein eigenes Dokument ergibt
korpus.saetze <- corpus_reshape(korpus, to = "sentences")
korpus.saetze[1] 01.1
"CHAPTER ONE DUDLEY DEMENTED he hottest day of the summer so far was drawing to a close and a drowsy silence lay over the large, square houses of Privet Drive." # Mit corpus_sample() ein zufälliges Sample aus dem Satz-Korpus ziehen
zufallssatz <- corpus_sample(korpus.saetze, size = 1)
zufallssatz[1] 33.46
"They walked on for what seemed a long time, until they were once again so deep into the forest that the dense tree canopy blocked out all light." # Korpus mit corpus_subset() nach Metadaten filtern (hier: Satzlänge) um ein Teilkorpus zu bilden, in dem nur längere Sätze enthalten sind
docvars(korpus.saetze, "Zeichenanzahl") <- ntoken(korpus.saetze)
docvars(korpus.saetze, "LangerSatz") <- ntoken(korpus.saetze)>=25
korpus.saetze_lang <- corpus_subset(korpus.saetze, LangerSatz == TRUE)
korpus.saetze_lang[1:3] 01.1
"CHAPTER ONE DUDLEY DEMENTED he hottest day of the summer so far was drawing to a close and a drowsy silence lay over the large, square houses of Privet Drive."
01.2
"Cars that were usually gleaming stood dusty in their drives and lawns that were once emerald green lay parched and yellowing; the use of hosepipes had been banned due to drought."
01.3
"Deprived of their usual car-washing and lawn-mowing pursuits, the inhabitants of Privet Drive had retreated into the shade of their cool houses, windows thrown wide in the hope of tempting in a nonexistent breeze." # Korpora lassen sich mit corpus_segment() auch nach bestimmten Kriterien aufspalten# Konkordanzen zu Suchtermen erstellen (auch "keywords-in-context" oder "KWIC" genannt)
konkordanz <- kwic(korpus, "magi*")
head(konkordanz)
[01, 2420] over his shoulder. Someone | magical | had been near him as
[01, 2479] Perhaps it hadn't been a | magical | sound after all. Perhaps
[01, 3582] completely cut off from the | magical | world, reduced to squatting
[01, 4224] . He must not use | magic | .... He
[01, 5823] Harry thought he had done | magic | without meaning to, despite
[02, 911] own ways of detecting underage | magic | , they'll know already, konkordanz <- kwic(korpus, phrase("Professor \\w+"), valuetype = "regex", case_insensitive = FALSE)
head(konkordanz)
[03, 1016:1017] lower his wand." | Professor Moody | ?" he said uncertainly
[03, 1199:1200] than a year." | P-Professor Lupin | ?" he said disbelievingly
[04, 6639:6640] favorite teacher at Hogwarts, | Professor Snape | . Harry leaned farther over
[06, 7957:7958] of his Transfiguration teacher, | Professor McGonagall | , looking very odd in
[06, 8491:8492] had answered it." | Professor Dumbledore | doesn't think it's a good
[08, 1522:1523] ?"" Yes, | Professor Lupin | taught me in my third konkordanz <- kwic(korpus, c("lov*", "hate*"), window = 10, case_insensitive = FALSE)
head(konkordanz, 30)
[01, 4157] ... let them try... He'd | love |
[01, 6894] in its slimy hands, prizing them slowly, almost | lovingly |
[04, 679] had seen her." Oh, Harry, it's | lovely |
[04, 4835] Harry let out a low whistle." Bet Percy | loved |
[04, 5789] a joke, who tells ridiculous tall stories because he | loves |
[04, 6125] " Crookshanks," said Ginny unblushingly." He | loves |
[05, 5887] of power now, and much more confident. He | loves |
[06, 3415] RUNNING A HIDEOUT FOR STOLEN GOODS!"" I | love |
[06, 4519] say if she saw Kreacher serving him, how she | hated |
[06, 5321] through his long, unkempt hair." Because I | hated |
[06, 6001] ." Andromeda's sisters are still here because they made | lovely |
[09, 4542] air of getting the worst over quickly." His | lovely |
[09, 5159] " 164 THE WOES OF MRS. WEASLEY" I'd | love |
[10, 4044] make Goyle do lines, it'll kill him, he | hates |
[11, 4012] ") and continued:" Well, it is | lovely |
[12, 3877] She had been Cedric's girlfriend; she could easily have | hated |
[12, 4763] front of the class all looked up eagerly; they | loved |
[13, 8135] that there might be another teacher in the world he | hated |
[14, 2423] called him brave.... She did not | hate |
[15, 6066] was whispering with Crabbe and Goyle; he would surely | love |
[17, 2107] to each other too." Oh, I've always | loved |
[17, 4608] raised. Snape and Umbridge, the two teachers he | hated |
[17, 5398] who's talking, you walked out of Divination, you | hate |
[17, 5409] Trelawney!" said Ron indignantly." I don't | hate |
[18, 1523] I mean... well, I think he'd | love |
[18, 3769] and rereading a passage about the uses of scurvy-grass, | lovage |
[19, 3736] close call, Alicia-- and the crowd are | loving |
[20, 4588] magic against' em, an' that's what giants | hate |
[23, 6089] .. He's the Trainee Healer, you know, | lovely |
[23, 7346] ." Oh Gilderoy, you've got visitors! How | lovely |
to vent some of his frustration on the boys who
apart, lowering its hooded head toward Dudley's face as
to see you!" she whispered, pulling him
that." Ron laughed in a hollow sort of
being famous and wants to keep it going."
playing with them."" Oh," said
being Minister of Magic, and he's managed to convince
hearing Mum shouting at someone else," said Fred
him, what a disappointment he was--"
the whole lot of them: my parents, with
, respectable pure-blood marriages, but Andromeda married a Muggleborn
shiny new prefect's badge." Fred's words took a
to see him try," sniggered George."
writing," said Ron happily. He lowered his
to be back at Hogwarts, I must say!
Harry for coming out of the Triwizard maze alive when
hearing Snape taunt Harry." The Draught of Peace
more than Snape, but as he walked back toward
him for being alive.... Of course
this opportunity to tell tales on Hagrid to a member
that owl, she's so beautiful," Harry heard
most... it was hard to decide which
Trelawney!" said Ron indignantly." I don't
her," said Hermione loftily." I just
to be forming secret defense societies right under the nose
, and sneezewort and not taking in a word of
this, just listen to them, what's that they're
abou' wizards. We had ter leg it an
young chap and very interested in... um
, and on Christmas Day too! Do you know In RStudio, the function View() can be used to look at a concordance.
#View(konkordanz)
# Speicherung von Konkordanzen
write_delim(konkordanz, path = "konkordanz.csv", delim = ";") # Datei ist Excel-kompatibel
# Plotten der Häufigkeit und Dispersion pro Kapitel von Tokens mit den Wortstämmen 'dark' und 'light'
term1 <- kwic(korpus, "dark", valuetype = "regex", case_insensitive = FALSE) %>% group_by(docname) %>% summarise(n = n()) %>% mutate(perc = n/(korpus.stats$Tokens/100), Suchterm = "dark")
term2 <- kwic(korpus, "light", valuetype = "regex", case_insensitive = FALSE) %>% group_by(docname) %>% summarise(n = n()) %>% mutate(perc = n/(korpus.stats$Tokens/100), Suchterm = "light")
# Absolute und relative Häufigkeit der Begriffe (Anmerkung: später verwendet man hierfür dfm() und dfm_weight())
ggplot(bind_rows(term1, term2), aes(docname, n, group = Suchterm, col = Suchterm)) + geom_line(size = 1) + ggtitle("Häufigkeit der Suchbegriffe \"dark\" und \"light\" pro Kapitel (absolut)") + xlab("Kapitel") + ylab("Wörter (gesamt)")
ggplot(bind_rows(term1, term2), aes(docname, perc, group = Suchterm, col = Suchterm)) + geom_line(size = 1) + ggtitle("Häufigkeit der Suchbegriffe \"dark\" und \"light\" pro Kapitel (relativ)") + xlab("Kapitel") + ylab("Wörter (%)")
# Dispersion der Begriffe als "xray-plot"
textplot_xray(kwic(korpus, "dark", valuetype = "regex", case_insensitive = FALSE))
textplot_xray(kwic(korpus, "light", valuetype = "regex", case_insensitive = FALSE))
# Ein Korpus tokenisiern (in einzelne Wörter oder N-Gramme aufspalten) und Begriffe behalten/entfernen
meine.tokens <- tokens(korpus)
head(meine.tokens$`01`, 10) [1] "CHAPTER" "ONE" "DUDLEY" "DEMENTED" "he" "hottest" "day" "of"
[9] "the" "summer" # N-Gramme extrahieren
meine.tokens <- tokens(korpus, ngrams = 2)
head(meine.tokens$`01`, 10) [1] "CHAPTER_ONE" "ONE_DUDLEY" "DUDLEY_DEMENTED" "DEMENTED_he" "he_hottest"
[6] "hottest_day" "day_of" "of_the" "the_summer" "summer_so" meine.tokens <- tokens(korpus, ngrams = 1:3)
tail(meine.tokens$`01`, 10) [1] "?_Oh_," "Oh_,_I'm" ",_I'm_going"
[4] "I'm_going_to" "going_to_kill" "to_kill_Mundungus"
[7] "kill_Mundungus_Fletcher" "Mundungus_Fletcher_!" "Fletcher_!_\""
[10] "!_\"_19" # Begriffe behalten/entfernen
meine.tokens <- tokens(korpus)
begriffe.behalten <- tokens_select(meine.tokens, c("hermione", "ron", "harry", "dumbledore")) # mit dem Parameter padding = TRUE bleiben Platzhalter zurück
head(begriffe.behalten$`01`, 10) [1] "Harry" "Harry" "Harry" "Harry" "Harry" "Harry" "Harry" "Harry" "Harry" "Harry"begriffe.entfernen <- tokens_remove(meine.tokens, c("CHAPTER", "ONE", "DUDLEY", "DEMENTED"))
head(begriffe.entfernen$`01`, 10) [1] "he" "hottest" "day" "of" "the" "summer" "so" "far" "was"
[10] "drawing"meine.tokens <- tokens(korpus, remove_numbers = TRUE, remove_punct = TRUE, remove_symbols = TRUE)
meine.tokens <- tokens_remove(meine.tokens, c(stopwords("english"), "harry", "chapter", "said"))
head(meine.tokens$`01`, 10) [1] "ONE" "DUDLEY" "DEMENTED" "hottest" "day" "summer" "far" "drawing"
[9] "close" "drowsy" # Extraktion und Speicherungvon Kollokationen mit textstat_collocations()
kollokationen <- textstat_collocations(meine.tokens, min_count = 25)
kollokationen collocation count count_nested length lambda z
1 mrs weasley 184 0 2 6.973555 48.664707
2 professor umbridge 167 0 2 5.095629 47.474295
3 mr weasley 161 0 2 6.868375 45.960426
4 fred george 124 0 2 6.337620 44.782618
5 ministry magic 64 0 2 6.615893 35.580119
6 professor mcgonagall 139 0 2 7.347825 33.821907
7 ron hermione 157 0 2 2.909281 31.992494
8 great hall 47 0 2 6.017124 31.076352
9 looked around 84 0 2 3.934905 30.933777
10 uncle vernon 85 0 2 10.551177 27.265419
11 defense dark 46 0 2 7.374528 27.160235
12 aunt petunia 56 0 2 10.441399 25.518916
13 years ago 25 0 2 6.840397 24.580525
14 professor trelawney 52 0 2 6.445972 24.408430
15 last night 32 0 2 5.058194 24.281962
16 entrance hall 38 0 2 8.112945 23.991674
17 common room 53 0 2 7.538538 23.593103
18 dark lord 27 0 2 6.364457 23.514270
19 shook head 28 0 2 6.172657 21.494582
20 crabbe goyle 27 0 2 10.504054 21.397893
21 mr potter 29 0 2 4.320014 20.880404
22 oh yeah 29 0 2 4.190146 20.386970
23 first time 29 0 2 3.923284 19.245503
24 daily prophet 41 0 2 11.570971 18.800701
25 come back 36 0 2 3.119727 17.341535
26 looking around 34 0 2 3.038782 16.638096
27 professor grubbly-plank 26 0 2 6.594373 16.631727
28 looked though 34 0 2 2.943839 16.172385
29 yeah well 26 0 2 3.177652 15.202403
30 grimmauld place 33 0 2 9.634946 14.547246
31 department mysteries 44 0 2 12.141734 14.134293
32 st mungo's 35 0 2 12.195470 13.995062
33 looked like 29 0 2 2.714233 13.942871
34 asked hermione 32 0 2 2.532200 13.496848
35 asked ron 29 0 2 2.445331 12.493816
36 looked back 29 0 2 2.276949 11.808958
37 mrs figg 40 0 2 9.608894 11.520842
38 hermione looking 29 0 2 2.030461 10.535466
39 ron looking 26 0 2 1.936661 9.566674
40 hermione looked 28 0 2 1.789375 9.192839
41 death eaters 82 0 2 12.527077 8.803062
42 dark arts 52 0 2 11.292038 7.935323
43 death eater 36 0 2 11.245627 7.882208write_delim(kollokationen, path = "kollokationen.csv", delim = ";") # Datei ist Excel-kompatibel# Kalkulation und Speicherung zahlreicher Lesbarkeitsindizes mit textstat_readability()
lesbarkeit <- textstat_readability(korpus)
head(lesbarkeit) document ARI ARI.simple Bormuth Bormuth.GP Coleman Coleman.C2 Coleman.Liau
1 01 6.836082 54.65364 -2.302520 31979180 50.45570 52.27434 53.35979
2 02 5.265785 51.53337 -1.757477 14502413 51.55891 55.62185 56.11683
3 03 6.023832 53.03881 -2.017620 21504372 52.40193 55.13809 54.66622
4 04 6.014858 53.02719 -2.041362 22379434 52.59436 55.20335 54.91114
5 05 6.105195 53.18658 -1.985803 20598650 49.49488 52.67736 54.01429
6 06 8.137777 57.25549 -2.815579 58237570 51.38829 51.55697 52.14571
Coleman.Liau.grade Coleman.Liau.short Dale.Chall Dale.Chall.old Dale.Chall.PSK Danielson.Bryan
1 8.443154 8.443738 -40.93566 20.14072 15.67541 5.279821
2 7.687713 7.688088 -39.08320 20.00755 15.51540 4.985606
3 8.085186 8.085671 -39.96436 20.07089 15.59151 5.128199
4 8.018077 8.018569 -40.04998 20.07705 15.59891 5.119277
5 8.263819 8.264296 -39.84528 20.06233 15.58123 5.158928
6 8.775817 8.776536 -42.70812 20.26813 15.82851 5.507609
Danielson.Bryan.2 Dickes.Steiwer DRP ELF Farr.Jenkins.Paterson Flesch Flesch.PSK
1 85.94569 -282.8818 330.2520 4.475490 -45.03049 73.98110 5.276539
2 86.20437 -243.6421 275.7477 3.540870 -42.29181 75.86899 5.112689
3 86.05450 -263.9449 301.7620 3.841962 -43.57756 76.80874 5.091788
4 86.21935 -262.6647 304.1362 3.859091 -43.70112 76.75940 5.097322
5 85.68072 -262.6659 298.5803 4.079812 -43.43843 74.53762 5.209929
6 86.13294 -314.5863 381.5579 5.151220 -47.62624 71.24271 5.483440
Flesch.Kincaid FOG FOG.PSK FOG.NRI FORCAST FORCAST.RGL Fucks Linsear.Write LIW
1 6.517690 8.544485 4.559415 480.3796 9.662128 9.058340 64.40441 4.009804 32.15270
2 5.587405 7.773799 3.911642 453.9707 9.533848 8.917233 51.82957 3.617391 29.41074
3 5.773582 7.730287 4.118882 347.4604 9.435822 8.809404 57.80926 3.291553 30.60928
4 5.811290 7.901016 4.182058 427.1935 9.413447 8.784791 58.16364 3.481818 29.97607
5 6.047485 7.896423 4.130299 396.9863 9.773851 9.181236 57.49765 3.549296 31.05767
6 7.537797 9.811609 5.309526 677.9953 9.553687 8.939056 75.95610 5.131707 35.43890
nWS nWS.2 nWS.3 nWS.4 RIX Scrabble SMOG SMOG.C SMOG.simple SMOG.de
1 4.277500 4.905644 3.691459 4.041801 2.556373 1.734938 8.848843 8.793500 8.483934 3.483934
2 3.805996 4.456597 3.404618 3.536736 2.073043 1.735528 8.561762 8.539936 8.208688 3.208688
3 3.576600 4.230881 3.237275 3.495649 2.288828 1.759631 8.311314 8.320418 7.968566 2.968566
4 3.699177 4.363460 3.350615 3.611677 2.211364 1.747241 8.458987 8.449651 8.110150 3.110150
5 3.955232 4.552777 3.378599 3.611137 2.338028 1.724889 8.510386 8.494769 8.159430 3.159430
6 4.740328 5.395540 4.358304 4.888443 3.134146 1.731309 9.599720 9.464796 9.203854 4.203854
Spache Spache.old Strain Traenkle.Bailer Traenkle.Bailer.2 Wheeler.Smith meanSentenceLength
1 10.60134 11.46933 6.037500 -329.0287 -231.9098 44.75490 14.39951
2 10.27649 11.09078 4.946609 -292.4407 -227.6259 35.40870 11.71478
3 10.43101 11.27085 5.382834 -311.4975 -231.9484 38.41962 12.99183
4 10.44603 11.28834 5.430682 -310.2404 -228.7579 38.59091 13.11591
5 10.41013 11.24651 5.422535 -311.9287 -235.7086 40.79812 12.81925
6 10.91216 11.83153 7.122439 -361.4017 -233.7889 51.51220 16.96829
meanWordSyllables
1 1.397617
2 1.407512
3 1.381082
4 1.380177
5 1.409998
6 1.399166write_delim(lesbarkeit, path = "lesbarkeit.csv", delim = ";") # Datei ist Excel-kompatibel# Erstellung einer Dokument-Feature-Matrix (DFM) mit dfm()
# Anmerkung: Hier wird implizit der Befehl tokens() angewandt, um bestimmte Features zu entfernen
meine.dfm <- dfm(korpus, remove_numbers = TRUE, remove_punct = TRUE, remove_symbols = TRUE, remove = c(stopwords("english"), "harry", "chapter", "said"))
meine.dfmDocument-feature matrix of: 38 documents, 12,849 features (88.2% sparse).ndoc(meine.dfm) # Anzahl Dokumente[1] 38nfeat(meine.dfm) # Anzahl Features[1] 12849head(docnames(meine.dfm)) # In der DFM enthaltene Dokumente [1] "01" "02" "03" "04" "05" "06"head(featnames(meine.dfm), 50) # Features in chronologischer Reihenfolge [1] "one" "dudley" "demented" "hottest" "day" "summer" "far"
[8] "drawing" "close" "drowsy" "silence" "lay" "large" "square"
[15] "houses" "privet" "drive" "cars" "usually" "gleaming" "stood"
[22] "dusty" "drives" "lawns" "emerald" "green" "parched" "yellowing"
[29] "use" "hosepipes" "banned" "due" "drought" "deprived" "usual"
[36] "car-washing" "lawn-mowing" "pursuits" "inhabitants" "retreated" "shade" "cool"
[43] "windows" "thrown" "wide" "hope" "tempting" "nonexistent" "breeze"
[50] "person" head(meine.dfm, nf = 10) # Features und Texte als Matrix in chronologischer ReihenfolgeDocument-feature matrix of: 6 documents, 10 features (41.7% sparse).
6 x 10 sparse Matrix of class "dfm"
features
docs one dudley demented hottest day summer far drawing close drowsy
01 18 57 10 1 3 6 2 2 1 1
02 14 40 0 0 0 0 1 0 0 0
03 8 1 0 0 2 2 3 0 3 0
04 7 0 0 0 0 2 2 1 3 0
05 10 1 0 0 0 1 4 1 0 0
06 3 0 0 0 4 2 5 7 2 0topfeatures(meine.dfm) # Features nach Frequenz hermione ron back now just well professor sirius dumbledore
1218 1191 791 675 644 642 592 589 565
around
549 worthaeufigkeiten <- textstat_frequency(meine.dfm) # Worthäufigkeiten
head(worthaeufigkeiten) feature frequency rank docfreq group
1 hermione 1218 1 34 all
2 ron 1191 2 35 all
3 back 791 3 38 all
4 now 675 4 38 all
5 just 644 5 38 all
6 well 642 6 38 all# Sortieren der DFM nach Dokument- und Feature-Frequenzen
head(dfm_sort(meine.dfm, decreasing = TRUE, margin = "both"), nf = 10) Document-feature matrix of: 6 documents, 10 features (1.67% sparse).
6 x 10 sparse Matrix of class "dfm"
features
docs hermione ron back now just well professor sirius dumbledore around
13 53 66 39 31 34 18 24 1 15 14
12 64 64 19 20 19 21 89 2 6 13
30 86 21 33 33 17 29 1 3 4 19
22 0 21 18 19 15 14 21 28 56 18
26 41 36 27 15 25 20 35 1 13 18
28 22 15 21 29 17 17 16 27 9 19# Wortstammreduktion von Wörtern (derzeit nur für English)
meine.dfm.stemmed <- dfm_wordstem(meine.dfm)
topfeatures(meine.dfm.stemmed) look ron hermion back go now dumbledor sirius well just
1395 1306 1305 807 676 675 665 645 644 644 # Gewichten der DFM nach relativen Wortfrequenzen und TF-IDF
# Achtung: Gewichtung funktioniert immer aufgrund der Wort-Text-Relation, weshalb topfeatures() in Kombination mit dfm_weight() merkwürdige Resultate produziert
# Relative Frequenzen und TF-IDF sind nur kontrastiv sinnvoll (hier: für 1. Kapitel), da für das gesamte Korpus relative Frequenz == absolute Frequenz
meine.dfm.proportional <- dfm_weight(meine.dfm, scheme = "prop")
topfeatures(meine.dfm) # absolute Frequenzen für das gesamte Korpus hermione ron back now just well professor sirius dumbledore
1218 1191 791 675 644 642 592 589 565
around
549 topfeatures(meine.dfm.proportional) # ...ergibt wenig Sinn hermione ron back now just well dumbledore sirius around
0.3511613 0.3350604 0.2318582 0.1962570 0.1845747 0.1837653 0.1762414 0.1746589 0.1633448
professor
0.1587047 topfeatures(meine.dfm.proportional[1,]) # ...ergibt mehr Sinn dudley uncle news know vernon one aunt wand
0.019387755 0.008503401 0.006802721 0.006802721 0.006462585 0.006122449 0.006122449 0.006122449
around petunia
0.005442177 0.004761905 meine.dfm.propmax <- dfm_weight(meine.dfm, scheme = "propmax")
topfeatures(meine.dfm.propmax[1,]) dudley uncle news know vernon one aunt wand around petunia
1.0000000 0.4385965 0.3508772 0.3508772 0.3333333 0.3157895 0.3157895 0.3157895 0.2807018 0.2456140 meine.dfm.tfidf <- dfm_tfidf(meine.dfm)
topfeatures(meine.dfm.tfidf[2,]) vernon petunia uncle dudley aunt figg peck diddy dudley's owls
48.899573 40.517425 32.841846 25.021643 23.027708 22.053247 20.537187 11.058485 9.777236 9.015927 # Erstellung einer reduzierten Dokument-Feature-Matrix mit dfm_trim()
meine.dfm.trim <- dfm_trim(meine.dfm, min_count = 0.9) # Behalte nur die Features im 90. Häufigkeitsperzentil (=Top 10% alles Features nach Frequenz)min_count is deprecated, use min_termfrequse termfreq_type = 'prop' for fractional term frequencymeine.dfm.trimDocument-feature matrix of: 38 documents, 12,849 features (88.2% sparse).head(meine.dfm.trim, nf = 10)Document-feature matrix of: 6 documents, 10 features (41.7% sparse).
6 x 10 sparse Matrix of class "dfm"
features
docs one dudley demented hottest day summer far drawing close drowsy
01 18 57 10 1 3 6 2 2 1 1
02 14 40 0 0 0 0 1 0 0 0
03 8 1 0 0 2 2 3 0 3 0
04 7 0 0 0 0 2 2 1 3 0
05 10 1 0 0 0 1 4 1 0 0
06 3 0 0 0 4 2 5 7 2 0# Plotten einer Wortwolke der häufigsten Begriff
if (require("RColorBrewer")) textplot_wordcloud(meine.dfm, max.words = 25, colors = brewer.pal(8, "Set2"), scale = c(5,1))Lade nötiges Paket: RColorBrewer
colorsscalemax.words is deprecated; use colormin_size and max_sizemax_words instead
# Berechnung Wortähnlichkeiten
aehnlichkeit_woerter <- textstat_simil(meine.dfm, "voldemort", margin = "features", method = "cosine")
# Optionen sind: "correlation", "cosine", "jaccard", "eJaccard", "dice", "eDice", "simple matching", "hamann", "faith"
head(aehnlichkeit_woerter[order(aehnlichkeit_woerter[,1], decreasing = T),]) voldemort mattered servant destroying caring kill
1.0000000 0.8919566 0.8814420 0.8814420 0.8814420 0.8290403 # Berechnung von Wortdistanzen
distanz_woerter <- textstat_dist(meine.dfm, "voldemort", margin = "features", method = "minkowski")
# Optionen sind: "euclidean", "Chisquared", "Chisquared2", "hamming", "kullback". "manhattan", "maximum", "canberra", "minkowski"
head(distanz_woerter[order(distanz_woerter[,1], decreasing = T),]) hermione ron hagrid professor weasley umbridge
231.9806 228.3594 177.2879 168.2647 135.4880 133.1090 # Plotten der Kapitelnähe via Kosinusähnlichkeit (hier für Kapitel 1)
aehnlichkeit_kapitel <- data.frame(Text = factor(korpus.stats$Text, levels = rev(korpus.stats$Text)), as.matrix(textstat_simil(meine.dfm, selection = "01", margin = "documents", method = "cosine")))
ggplot(aehnlichkeit_kapitel, aes(X01, Text)) + geom_point(size = 2.5) + ggtitle("Kapitelnähe via Kosinusähnlichkeit (hier für Kapitel 1)") + xlab("Kosinunsähnlichkeit")
# Berechnung zahlreicher Metriken für die lexikalische Diversität des Kapitel mit textstat_lexdiv()
lexdiversitaet <- textstat_lexdiv(meine.dfm)
head(lexdiversitaet) document TTR C R CTTR U S Maas lgV0 lgeV0
1 01 0.4986395 0.9128653 27.03711 19.11813 39.80444 0.9266957 0.1585018 7.755174 17.85695
2 02 0.4272510 0.8952460 24.74370 17.49644 33.65564 0.9121800 0.1723737 7.083579 16.31054
3 03 0.5161426 0.9149574 25.20658 17.82374 39.71526 0.9269777 0.1586797 7.657676 17.63245
4 04 0.4597660 0.9021960 24.41959 17.26726 35.27876 0.9168961 0.1683617 7.217136 16.61807
5 05 0.4630524 0.9025228 24.02970 16.99156 35.19013 0.9167946 0.1685736 7.188954 16.55318
6 06 0.4656708 0.9062821 27.47458 19.42746 37.79112 0.9221857 0.1626691 7.594004 17.48584write_delim(lexdiversitaet, path = "lexdiversitaet.csv", delim = ";") # Datei ist Excel-kompatibel# Keyness nach Text (hier für Kapitel 01, 03, 26, 34)
keyness <- textstat_keyness(meine.dfm, target = "01", measure = "lr")
textplot_keyness(keyness)
keyness <- textstat_keyness(meine.dfm, target = "03", measure = "lr")
textplot_keyness(keyness)
keyness <- textstat_keyness(meine.dfm, target = "26", measure = "lr")
textplot_keyness(keyness)
keyness <- textstat_keyness(meine.dfm, target = "34", measure = "lr")
textplot_keyness(keyness)
# abgeschnittene labels sichtbar machen:
# textplot_keyness(keyness) + scale_y_continuous(limits = c(-75, 130))LS0tCnRpdGxlOiAiQWR2YW5jZWQgVGV4dCBNaW5pbmcgd2l0aCBSIGFuZCBxdWFudGVkYSIKYXV0aG9yOiAiQ29ybmVsaXVzIFB1c2NobWFubiwgQGNicHVzY2htYW5uIgpkYXRlOiAiTWF5IDE2dGgsIDIwMTgiCm91dHB1dDogaHRtbF9ub3RlYm9vawotLS0KCk1aRVMsIFVuaXZlcnNpdHkgb2YgTWFubmhlaW0KCiMgT3ZlcnZpZXcKCjEuIFdoeSBkbyB0ZXh0IG1pbmluZy9hdXRvbWF0ZWQgY29udGVudCBhbmFseXNpcz8KMi4gV2h5IHVzZSBxdWFudGVkYT8KMy4gR2V0dGluZyBkYXRhIGludG8gcXVhbnRlZGEgd2l0aCByZWFkdGV4dAo0LiBDb21waWxpbmcgYW5kIHdvcmtpbmcgd2l0aCBjb3Jwb3JhCjUuIE1lYXN1cmVzIGFuZCBtZXRyaWNzIGZvciB3b3JkcyBhbmQgdGV4dHMKNi4gQXBwbHlpbmcgbGV4aWNvbnMKNy4gVG9waWMgbW9kZWxpbmcKOC4gQXV0b21hdGluZyBjb250ZW50IGFuYWx5c2lzCgpEb3dubG9hZCB0aGUgZGF0YSBhbmQgbGV4aWNvbnMgW2hlcmVdKGh0dHA6Ly9ybWFya2Rvd24ucnN0dWRpby5jb20pLgoKIyBDb21waWxpbmcgYW5kIHdvcmtpbmcgd2l0aCBjb3Jwb3JhCkJlZm9yZSBnZXR0aW5nIHN0YXJ0ZWQsIHdlIGluc3RhbGwgYW5kIGxvYWQgdGhlIHJlcXVpcmVkIGxpYnJhcmllcy4KYGBge3J9CmlmKCFyZXF1aXJlKCJyZWFkdGV4dCIpKSBpbnN0YWxsLnBhY2thZ2VzKCJyZWFkdGV4dCIpCmlmKCFyZXF1aXJlKCJxdWFudGVkYSIpKSBpbnN0YWxsLnBhY2thZ2VzKCJxdWFudGVkYSIpCmlmKCFyZXF1aXJlKCJsdWJyaWRhdGUiKSkgaW5zdGFsbC5wYWNrYWdlcygibHVicmlkYXRlIikKaWYoIXJlcXVpcmUoInRpZHl2ZXJzZSIpKSBpbnN0YWxsLnBhY2thZ2VzKCJ0aWR5dmVyc2UiKQpsaWJyYXJ5KCJyZWFkdGV4dCIpCmxpYnJhcnkoInF1YW50ZWRhIikKbGlicmFyeSgibHVicmlkYXRlIikKbGlicmFyeSgidGlkeXZlcnNlIikKdGhlbWVfc2V0KHRoZW1lX2J3KCkpCmBgYApXZSBzdGFydCBvdXIgYW5hbHlzaXMgYnkgcmVhZGluZyBpbiB0aGUgSGFycnkgUG90dGVyIGNvcnB1cyBhbmQgYXNzaWduaW5nIGJhc2ljIG1ldGEgZGF0YS4gV2UgdXNlIHRoZSBgcmVhZHRleHQoKWAgZnVuY3Rpb24gdG8gbG9hZCB0aGUgZGF0YSBhbmQgdGhlIGBjb3JwdXMoKWAgZnVuY3Rpb24gdG8gZ2VuZXJhdGUgcXVhbnRlZGEgY29ycHVzIGZyb20gaXQuIApgYGB7cn0KIyBFaW5sZXNlbiBkZXIgSGFycnkgUG90dGVyLVRleHRkYXRlaWVuIG1pdCByZWFkdGV4dCgpIHVuZCBFcnN0ZWxsdW5nIGVpbmVzIHF1YW50ZWRhLUtvcnB1cyBuZWJzdCB2b24gTWV0YWRhdGVuIG1pdCBjb3JwdXMoKQpxdWVsbGRhdGVpZW4gPC0gZGlyKCJkYXRhL2hhcnJ5cG90dGVyIiwgcGF0dGVybiA9ICJjaGFwdGVyX1swLTldezJ9LnR4dCIpCmRhdGVuLmhhcnJ5cG90dGVyIDwtIHJlYWR0ZXh0KHBhc3RlKCJkYXRhL2hhcnJ5cG90dGVyIiwgcXVlbGxkYXRlaWVuLCBzZXAgPSAiLyIpKQpkYXRlbi5oYXJyeXBvdHRlciRkb2NfaWQgPC0gc3ByaW50ZigiJTAyZCIsIDE6MzgpICMgcHJha3Rpc2NoZXJlIEJlbmVubnVuZyBkZXIgS2FwaXRlbAprb3JwdXMgPC0gY29ycHVzKGRhdGVuLmhhcnJ5cG90dGVyLCBkb2NpZF9maWVsZCA9ICJkb2NfaWQiKQprb3JwdXMuc3RhdHMgPC0gc3VtbWFyeShrb3JwdXMsIG4gPSAxMDAwMDAwKQpgYGAKTmV4dCB3ZSBnZW5lcmF0ZSBhIHNldCBvZiBiYXNpYyBjb3JwdXMgc3RhdGlzdGljcyB3aXRoIHRoZSBmdW5jdGlvbnMgYG5kb2MoKWAsIGBudG9rZW4oKWAsIGBudHlvcGUoKWAgYW5kIGBuc2VudGVuY2UoKWAuIApgYGB7cn0Ka29ycHVzCm5kb2Moa29ycHVzKSAjIG51bWJlciBvZiBkb2N1bWVudHMKbnRva2VuKGtvcnB1cykgIyBudW1iZXIgb2YgdG9rZW5zIApudHlwZShrb3JwdXMpICMgbnVtYmVyIG9mIHR5cGVzCm5zZW50ZW5jZShrb3JwdXMpICMgbnVtYmVyIG9mIHNlbnRlbmNlcwprb3JwdXMuc3RhdHMKYGBgCkxldCdzIGV4YW1pbmUgKGEpIHRoZSBudW1iZXIgb2YgdG9rZW5zIHBlciBjaGFwdGVyLCAoYikgdGhlIG51bWJlciBvZiB0eXBlcyBwZXIgY2hhcHRlciBhbmQgKGMpIHRoZSBudW1iZXIgb2Ygc2VudGVuY2VzIHBlciBjaGFwdGVyLiAKYGBge3J9CiMgUGxvdHRlbiB2b24gKGEpIFRva2VucyBwcm8gS2FwaXRlbCAoYikgVHlwZW4gcHJvIEthcGl0ZWwgKGMpIFPDpHR6ZSBwcm8gS2FwaXRlbApnZ3Bsb3Qoa29ycHVzLnN0YXRzLCBhZXMoVGV4dCwgVG9rZW5zLCBncm91cD0xKSkgKyBnZW9tX2xpbmUoKSArIGdlb21fcG9pbnQoKSArIGdndGl0bGUoIlRva2VucyBwcm8gS2FwaXRlbCIpCmdncGxvdChrb3JwdXMuc3RhdHMsIGFlcyhUZXh0LCBUeXBlcywgZ3JvdXA9MSkpICsgZ2VvbV9saW5lKCkgKyBnZW9tX3BvaW50KCkgKyBnZ3RpdGxlKCJUeXBlcyBwcm8gS2FwaXRlbCIpCmdncGxvdChrb3JwdXMuc3RhdHMsIGFlcyhUZXh0LCBTZW50ZW5jZXMsIGdyb3VwPTEpKSArIGdlb21fbGluZSgpICsgZ2VvbV9wb2ludCgpICsgZ2d0aXRsZSgiU8OkdHplIHBybyBLYXBpdGVsIikKCmBgYAoKYGBge3J9CiMgLi4udW5kIHp1c2FtbWVuCmdncGxvdChrb3JwdXMuc3RhdHMgJT4lIGdhdGhlcihUeXBlcywgVG9rZW5zLCBTZW50ZW5jZXMsIGtleSA9ICJFaW5oZWl0IiwgdmFsdWUgPSAiQW56YWhsIiksIGFlcyhUZXh0LCBBbnphaGwsIGdyb3VwID0gRWluaGVpdCwgY29sID0gRWluaGVpdCkpICsgZ2VvbV9saW5lKHNpemUgPSAxKSArIGdndGl0bGUoIlRva2VucywgVHlwZXMgdW5kIFPDpHR6ZSBwcm8gS2FwaXRlbCIpCmBgYApQbG90dGVpbmcgdGhlIHR5cGUtdG9rZW4gcmF0aW8gKFRUUikKYGBge3J9CmdncGxvdChrb3JwdXMuc3RhdHMsIGFlcyhUb2tlbnMsIFR5cGVzLCBncm91cD0xLCBsYWJlbCA9IFRleHQpKSArIGdlb21fc21vb3RoKG1ldGhvZCA9ICJsbSIsIHNlID0gRkFMU0UpICsgZ2VvbV90ZXh0KGNoZWNrX292ZXJsYXAgPSBUKSArIGdndGl0bGUoIlR5cGUtVG9rZW4gcmF0aW8gcGVyIGNoYXB0ZXIiKQpgYGAKCgpgYGB7cn0KIyBLb3Jwb3JhIGZpbHRlcm4sIHVtZm9ybWVuLCB1bmQgbWl0IE1ldGFkYXRlbiB2ZXJzZWhlbgojIEluaGFsdCBkZXMgZXJzdGVuIEthcGl0ZWxzIHdpZWRlcmdlYmVuCmtvcnB1c1sxXQojIEtvcnB1cyBtaXQgY29ycHVzX3Jlc2hhcGUgdW1mb3JtZW4sIHNvIGRhc3MgamVkZXIgU2F0eiBlaW4gZWlnZW5lcyBEb2t1bWVudCBlcmdpYnQKa29ycHVzLnNhZXR6ZSA8LSBjb3JwdXNfcmVzaGFwZShrb3JwdXMsIHRvID0gInNlbnRlbmNlcyIpCmtvcnB1cy5zYWV0emVbMV0KIyBNaXQgY29ycHVzX3NhbXBsZSgpIGVpbiB6dWbDpGxsaWdlcyBTYW1wbGUgYXVzIGRlbSBTYXR6LUtvcnB1cyB6aWVoZW4KenVmYWxsc3NhdHogPC0gY29ycHVzX3NhbXBsZShrb3JwdXMuc2FldHplLCBzaXplID0gMSkKenVmYWxsc3NhdHpbMV0KIyBLb3JwdXMgbWl0IGNvcnB1c19zdWJzZXQoKSBuYWNoIE1ldGFkYXRlbiBmaWx0ZXJuIChoaWVyOiBTYXR6bMOkbmdlKSB1bSBlaW4gVGVpbGtvcnB1cyB6dSBiaWxkZW4sIGluIGRlbSBudXIgbMOkbmdlcmUgU8OkdHplIGVudGhhbHRlbiBzaW5kCmRvY3ZhcnMoa29ycHVzLnNhZXR6ZSwgIlplaWNoZW5hbnphaGwiKSA8LSBudG9rZW4oa29ycHVzLnNhZXR6ZSkKZG9jdmFycyhrb3JwdXMuc2FldHplLCAiTGFuZ2VyU2F0eiIpIDwtIG50b2tlbihrb3JwdXMuc2FldHplKT49MjUKa29ycHVzLnNhZXR6ZV9sYW5nIDwtIGNvcnB1c19zdWJzZXQoa29ycHVzLnNhZXR6ZSwgTGFuZ2VyU2F0eiA9PSBUUlVFKQprb3JwdXMuc2FldHplX2xhbmdbMTozXQojIEtvcnBvcmEgbGFzc2VuIHNpY2ggbWl0IGNvcnB1c19zZWdtZW50KCkgYXVjaCBuYWNoIGJlc3RpbW10ZW4gS3JpdGVyaWVuIGF1ZnNwYWx0ZW4KYGBgCgpgYGB7cn0KIyBLb25rb3JkYW56ZW4genUgU3VjaHRlcm1lbiBlcnN0ZWxsZW4gKGF1Y2ggImtleXdvcmRzLWluLWNvbnRleHQiIG9kZXIgIktXSUMiIGdlbmFubnQpCmtvbmtvcmRhbnogPC0ga3dpYyhrb3JwdXMsICJtYWdpKiIpCmhlYWQoa29ua29yZGFueikKa29ua29yZGFueiA8LSBrd2ljKGtvcnB1cywgcGhyYXNlKCJQcm9mZXNzb3IgXFx3KyIpLCB2YWx1ZXR5cGUgPSAicmVnZXgiLCBjYXNlX2luc2Vuc2l0aXZlID0gRkFMU0UpCmhlYWQoa29ua29yZGFueikKa29ua29yZGFueiA8LSBrd2ljKGtvcnB1cywgYygibG92KiIsICJoYXRlKiIpLCB3aW5kb3cgPSAxMCwgY2FzZV9pbnNlbnNpdGl2ZSA9IEZBTFNFKQpoZWFkKGtvbmtvcmRhbnosIDMwKQpgYGAKSW4gUlN0dWRpbywgdGhlIGZ1bmN0aW9uIGBWaWV3KClgIGNhbiBiZSB1c2VkIHRvIGxvb2sgYXQgYSBjb25jb3JkYW5jZS4gCmBgYHtyfQojVmlldyhrb25rb3JkYW56KQojIFNwZWljaGVydW5nIHZvbiBLb25rb3JkYW56ZW4Kd3JpdGVfZGVsaW0oa29ua29yZGFueiwgcGF0aCA9ICJrb25rb3JkYW56LmNzdiIsIGRlbGltID0gIjsiKSAjIERhdGVpIGlzdCBFeGNlbC1rb21wYXRpYmVsCiMgUGxvdHRlbiBkZXIgSMOkdWZpZ2tlaXQgdW5kIERpc3BlcnNpb24gcHJvIEthcGl0ZWwgdm9uIFRva2VucyBtaXQgZGVuIFdvcnRzdMOkbW1lbiAnZGFyaycgdW5kICdsaWdodCcKdGVybTEgPC0ga3dpYyhrb3JwdXMsICJkYXJrIiwgdmFsdWV0eXBlID0gInJlZ2V4IiwgY2FzZV9pbnNlbnNpdGl2ZSA9IEZBTFNFKSAlPiUgZ3JvdXBfYnkoZG9jbmFtZSkgJT4lIHN1bW1hcmlzZShuID0gbigpKSAlPiUgbXV0YXRlKHBlcmMgPSBuLyhrb3JwdXMuc3RhdHMkVG9rZW5zLzEwMCksIFN1Y2h0ZXJtID0gImRhcmsiKSAKdGVybTIgPC0ga3dpYyhrb3JwdXMsICJsaWdodCIsIHZhbHVldHlwZSA9ICJyZWdleCIsIGNhc2VfaW5zZW5zaXRpdmUgPSBGQUxTRSkgJT4lIGdyb3VwX2J5KGRvY25hbWUpICU+JSBzdW1tYXJpc2UobiA9IG4oKSkgJT4lIG11dGF0ZShwZXJjID0gbi8oa29ycHVzLnN0YXRzJFRva2Vucy8xMDApLCBTdWNodGVybSA9ICJsaWdodCIpIAojIEFic29sdXRlIHVuZCByZWxhdGl2ZSBIw6R1Zmlna2VpdCBkZXIgQmVncmlmZmUgKEFubWVya3VuZzogc3DDpHRlciB2ZXJ3ZW5kZXQgbWFuIGhpZXJmw7xyIGRmbSgpIHVuZCBkZm1fd2VpZ2h0KCkpCmdncGxvdChiaW5kX3Jvd3ModGVybTEsIHRlcm0yKSwgYWVzKGRvY25hbWUsIG4sIGdyb3VwID0gU3VjaHRlcm0sIGNvbCA9IFN1Y2h0ZXJtKSkgKyBnZW9tX2xpbmUoc2l6ZSA9IDEpICsgZ2d0aXRsZSgiSMOkdWZpZ2tlaXQgZGVyIFN1Y2hiZWdyaWZmZSBcImRhcmtcIiB1bmQgXCJsaWdodFwiIHBybyBLYXBpdGVsIChhYnNvbHV0KSIpICsgeGxhYigiS2FwaXRlbCIpICsgeWxhYigiV8O2cnRlciAoZ2VzYW10KSIpCmdncGxvdChiaW5kX3Jvd3ModGVybTEsIHRlcm0yKSwgYWVzKGRvY25hbWUsIHBlcmMsIGdyb3VwID0gU3VjaHRlcm0sIGNvbCA9IFN1Y2h0ZXJtKSkgKyBnZW9tX2xpbmUoc2l6ZSA9IDEpICsgZ2d0aXRsZSgiSMOkdWZpZ2tlaXQgZGVyIFN1Y2hiZWdyaWZmZSBcImRhcmtcIiB1bmQgXCJsaWdodFwiIHBybyBLYXBpdGVsIChyZWxhdGl2KSIpICsgeGxhYigiS2FwaXRlbCIpICsgeWxhYigiV8O2cnRlciAoJSkiKQojIERpc3BlcnNpb24gZGVyIEJlZ3JpZmZlIGFscyAieHJheS1wbG90Igp0ZXh0cGxvdF94cmF5KGt3aWMoa29ycHVzLCAiZGFyayIsIHZhbHVldHlwZSA9ICJyZWdleCIsIGNhc2VfaW5zZW5zaXRpdmUgPSBGQUxTRSkpCnRleHRwbG90X3hyYXkoa3dpYyhrb3JwdXMsICJsaWdodCIsIHZhbHVldHlwZSA9ICJyZWdleCIsIGNhc2VfaW5zZW5zaXRpdmUgPSBGQUxTRSkpCmBgYAoKYGBge3J9CiMgRWluIEtvcnB1cyB0b2tlbmlzaWVybiAoaW4gZWluemVsbmUgV8O2cnRlciBvZGVyIE4tR3JhbW1lIGF1ZnNwYWx0ZW4pIHVuZCBCZWdyaWZmZSBiZWhhbHRlbi9lbnRmZXJuZW4KbWVpbmUudG9rZW5zIDwtIHRva2Vucyhrb3JwdXMpCmhlYWQobWVpbmUudG9rZW5zJGAwMWAsIDEwKQojIE4tR3JhbW1lIGV4dHJhaGllcmVuCm1laW5lLnRva2VucyA8LSB0b2tlbnMoa29ycHVzLCBuZ3JhbXMgPSAyKQpoZWFkKG1laW5lLnRva2VucyRgMDFgLCAxMCkKbWVpbmUudG9rZW5zIDwtIHRva2Vucyhrb3JwdXMsIG5ncmFtcyA9IDE6MykKdGFpbChtZWluZS50b2tlbnMkYDAxYCwgMTApCiMgQmVncmlmZmUgYmVoYWx0ZW4vZW50ZmVybmVuCm1laW5lLnRva2VucyA8LSB0b2tlbnMoa29ycHVzKQpiZWdyaWZmZS5iZWhhbHRlbiA8LSB0b2tlbnNfc2VsZWN0KG1laW5lLnRva2VucywgYygiaGVybWlvbmUiLCAicm9uIiwgImhhcnJ5IiwgImR1bWJsZWRvcmUiKSkgIyBtaXQgZGVtIFBhcmFtZXRlciBwYWRkaW5nID0gVFJVRSBibGVpYmVuIFBsYXR6aGFsdGVyIHp1csO8Y2sKaGVhZChiZWdyaWZmZS5iZWhhbHRlbiRgMDFgLCAxMCkKYmVncmlmZmUuZW50ZmVybmVuIDwtIHRva2Vuc19yZW1vdmUobWVpbmUudG9rZW5zLCBjKCJDSEFQVEVSIiwgIk9ORSIsICJEVURMRVkiLCAiREVNRU5URUQiKSkKaGVhZChiZWdyaWZmZS5lbnRmZXJuZW4kYDAxYCwgMTApCm1laW5lLnRva2VucyA8LSB0b2tlbnMoa29ycHVzLCByZW1vdmVfbnVtYmVycyA9IFRSVUUsIHJlbW92ZV9wdW5jdCA9IFRSVUUsIHJlbW92ZV9zeW1ib2xzID0gVFJVRSkKbWVpbmUudG9rZW5zIDwtIHRva2Vuc19yZW1vdmUobWVpbmUudG9rZW5zLCBjKHN0b3B3b3JkcygiZW5nbGlzaCIpLCAiaGFycnkiLCAiY2hhcHRlciIsICJzYWlkIikpCmhlYWQobWVpbmUudG9rZW5zJGAwMWAsIDEwKQpgYGAKCmBgYHtyfQojIEV4dHJha3Rpb24gdW5kIFNwZWljaGVydW5ndm9uIEtvbGxva2F0aW9uZW4gbWl0IHRleHRzdGF0X2NvbGxvY2F0aW9ucygpCmtvbGxva2F0aW9uZW4gPC0gdGV4dHN0YXRfY29sbG9jYXRpb25zKG1laW5lLnRva2VucywgbWluX2NvdW50ID0gMjUpCmtvbGxva2F0aW9uZW4Kd3JpdGVfZGVsaW0oa29sbG9rYXRpb25lbiwgcGF0aCA9ICJrb2xsb2thdGlvbmVuLmNzdiIsIGRlbGltID0gIjsiKSAjIERhdGVpIGlzdCBFeGNlbC1rb21wYXRpYmVsCmBgYAoKYGBge3J9CiMgS2Fsa3VsYXRpb24gdW5kIFNwZWljaGVydW5nIHphaGxyZWljaGVyIExlc2JhcmtlaXRzaW5kaXplcyBtaXQgdGV4dHN0YXRfcmVhZGFiaWxpdHkoKQpsZXNiYXJrZWl0IDwtIHRleHRzdGF0X3JlYWRhYmlsaXR5KGtvcnB1cykKaGVhZChsZXNiYXJrZWl0KQp3cml0ZV9kZWxpbShsZXNiYXJrZWl0LCBwYXRoID0gImxlc2JhcmtlaXQuY3N2IiwgZGVsaW0gPSAiOyIpICMgRGF0ZWkgaXN0IEV4Y2VsLWtvbXBhdGliZWwKYGBgCgpgYGB7cn0KIyBFcnN0ZWxsdW5nIGVpbmVyIERva3VtZW50LUZlYXR1cmUtTWF0cml4IChERk0pIG1pdCBkZm0oKQojIEFubWVya3VuZzogSGllciB3aXJkIGltcGxpeml0IGRlciBCZWZlaGwgdG9rZW5zKCkgYW5nZXdhbmR0LCB1bSBiZXN0aW1tdGUgRmVhdHVyZXMgenUgZW50ZmVybmVuCm1laW5lLmRmbSA8LSBkZm0oa29ycHVzLCByZW1vdmVfbnVtYmVycyA9IFRSVUUsIHJlbW92ZV9wdW5jdCA9IFRSVUUsIHJlbW92ZV9zeW1ib2xzID0gVFJVRSwgcmVtb3ZlID0gYyhzdG9wd29yZHMoImVuZ2xpc2giKSwgImhhcnJ5IiwgImNoYXB0ZXIiLCAic2FpZCIpKQptZWluZS5kZm0KbmRvYyhtZWluZS5kZm0pICMgQW56YWhsIERva3VtZW50ZQpuZmVhdChtZWluZS5kZm0pICMgQW56YWhsIEZlYXR1cmVzCmhlYWQoZG9jbmFtZXMobWVpbmUuZGZtKSkgIyBJbiBkZXIgREZNIGVudGhhbHRlbmUgRG9rdW1lbnRlIApoZWFkKGZlYXRuYW1lcyhtZWluZS5kZm0pLCA1MCkgIyBGZWF0dXJlcyBpbiBjaHJvbm9sb2dpc2NoZXIgUmVpaGVuZm9sZ2UKaGVhZChtZWluZS5kZm0sIG5mID0gMTApICMgRmVhdHVyZXMgdW5kIFRleHRlIGFscyBNYXRyaXggaW4gY2hyb25vbG9naXNjaGVyIFJlaWhlbmZvbGdlCnRvcGZlYXR1cmVzKG1laW5lLmRmbSkgIyBGZWF0dXJlcyBuYWNoIEZyZXF1ZW56CndvcnRoYWV1Zmlna2VpdGVuIDwtIHRleHRzdGF0X2ZyZXF1ZW5jeShtZWluZS5kZm0pICMgV29ydGjDpHVmaWdrZWl0ZW4KaGVhZCh3b3J0aGFldWZpZ2tlaXRlbikKYGBgCgpgYGB7cn0KIyBTb3J0aWVyZW4gZGVyIERGTSBuYWNoIERva3VtZW50LSB1bmQgRmVhdHVyZS1GcmVxdWVuemVuCmhlYWQoZGZtX3NvcnQobWVpbmUuZGZtLCBkZWNyZWFzaW5nID0gVFJVRSwgbWFyZ2luID0gImJvdGgiKSwgbmYgPSAxMCkgCmBgYAoKYGBge3J9CiMgV29ydHN0YW1tcmVkdWt0aW9uIHZvbiBXw7ZydGVybiAoZGVyemVpdCBudXIgZsO8ciBFbmdsaXNoKQptZWluZS5kZm0uc3RlbW1lZCA8LSBkZm1fd29yZHN0ZW0obWVpbmUuZGZtKQp0b3BmZWF0dXJlcyhtZWluZS5kZm0uc3RlbW1lZCkKYGBgCgpgYGB7cn0KIyBHZXdpY2h0ZW4gZGVyIERGTSBuYWNoIHJlbGF0aXZlbiBXb3J0ZnJlcXVlbnplbiB1bmQgVEYtSURGCiMgQWNodHVuZzogR2V3aWNodHVuZyBmdW5rdGlvbmllcnQgaW1tZXIgYXVmZ3J1bmQgZGVyIFdvcnQtVGV4dC1SZWxhdGlvbiwgd2VzaGFsYiB0b3BmZWF0dXJlcygpIGluIEtvbWJpbmF0aW9uIG1pdCBkZm1fd2VpZ2h0KCkgbWVya3fDvHJkaWdlIFJlc3VsdGF0ZSBwcm9kdXppZXJ0CiMgUmVsYXRpdmUgRnJlcXVlbnplbiB1bmQgVEYtSURGIHNpbmQgbnVyIGtvbnRyYXN0aXYgc2lubnZvbGwgKGhpZXI6IGbDvHIgMS4gS2FwaXRlbCksIGRhIGbDvHIgZGFzIGdlc2FtdGUgS29ycHVzIHJlbGF0aXZlIEZyZXF1ZW56ID09IGFic29sdXRlIEZyZXF1ZW56Cm1laW5lLmRmbS5wcm9wb3J0aW9uYWwgPC0gZGZtX3dlaWdodChtZWluZS5kZm0sIHNjaGVtZSA9ICJwcm9wIikKdG9wZmVhdHVyZXMobWVpbmUuZGZtKSAjIGFic29sdXRlIEZyZXF1ZW56ZW4gZsO8ciBkYXMgZ2VzYW10ZSBLb3JwdXMKdG9wZmVhdHVyZXMobWVpbmUuZGZtLnByb3BvcnRpb25hbCkgIyAuLi5lcmdpYnQgd2VuaWcgU2lubgp0b3BmZWF0dXJlcyhtZWluZS5kZm0ucHJvcG9ydGlvbmFsWzEsXSkgIyAuLi5lcmdpYnQgbWVociBTaW5uCm1laW5lLmRmbS5wcm9wbWF4IDwtIGRmbV93ZWlnaHQobWVpbmUuZGZtLCBzY2hlbWUgPSAicHJvcG1heCIpCnRvcGZlYXR1cmVzKG1laW5lLmRmbS5wcm9wbWF4WzEsXSkKbWVpbmUuZGZtLnRmaWRmIDwtIGRmbV90ZmlkZihtZWluZS5kZm0pCnRvcGZlYXR1cmVzKG1laW5lLmRmbS50ZmlkZlsyLF0pCmBgYAoKYGBge3J9CiMgRXJzdGVsbHVuZyBlaW5lciByZWR1emllcnRlbiBEb2t1bWVudC1GZWF0dXJlLU1hdHJpeCBtaXQgZGZtX3RyaW0oKQptZWluZS5kZm0udHJpbSA8LSBkZm1fdHJpbShtZWluZS5kZm0sIG1pbl9jb3VudCA9IDAuOSkgIyBCZWhhbHRlIG51ciBkaWUgRmVhdHVyZXMgaW0gOTAuIEjDpHVmaWdrZWl0c3BlcnplbnRpbCAoPVRvcCAxMCUgYWxsZXMgRmVhdHVyZXMgbmFjaCBGcmVxdWVueikKbWVpbmUuZGZtLnRyaW0KaGVhZChtZWluZS5kZm0udHJpbSwgbmYgPSAxMCkKYGBgCgoKYGBge3J9CiMgUGxvdHRlbiBlaW5lciBXb3J0d29sa2UgZGVyIGjDpHVmaWdzdGVuIEJlZ3JpZmYKaWYgKHJlcXVpcmUoIlJDb2xvckJyZXdlciIpKSB0ZXh0cGxvdF93b3JkY2xvdWQobWVpbmUuZGZtLCBtYXgud29yZHMgPSAyNSwgY29sb3JzID0gYnJld2VyLnBhbCg4LCAiU2V0MiIpLCBzY2FsZSA9IGMoNSwxKSkKYGBgCgpgYGB7cn0KIyBCZXJlY2hudW5nIFdvcnTDpGhubGljaGtlaXRlbgphZWhubGljaGtlaXRfd29lcnRlciA8LSB0ZXh0c3RhdF9zaW1pbChtZWluZS5kZm0sICJ2b2xkZW1vcnQiLCBtYXJnaW4gPSAiZmVhdHVyZXMiLCBtZXRob2QgPSAiY29zaW5lIikgCiMgT3B0aW9uZW4gc2luZDogImNvcnJlbGF0aW9uIiwgImNvc2luZSIsICJqYWNjYXJkIiwgImVKYWNjYXJkIiwgImRpY2UiLCAiZURpY2UiLCAic2ltcGxlIG1hdGNoaW5nIiwgImhhbWFubiIsICJmYWl0aCIKaGVhZChhZWhubGljaGtlaXRfd29lcnRlcltvcmRlcihhZWhubGljaGtlaXRfd29lcnRlclssMV0sIGRlY3JlYXNpbmcgPSBUKSxdKQpgYGAKCmBgYHtyfQojIEJlcmVjaG51bmcgdm9uIFdvcnRkaXN0YW56ZW4KZGlzdGFuel93b2VydGVyIDwtIHRleHRzdGF0X2Rpc3QobWVpbmUuZGZtLCAidm9sZGVtb3J0IiwgbWFyZ2luID0gImZlYXR1cmVzIiwgbWV0aG9kID0gIm1pbmtvd3NraSIpCiMgT3B0aW9uZW4gc2luZDogImV1Y2xpZGVhbiIsICJDaGlzcXVhcmVkIiwgIkNoaXNxdWFyZWQyIiwgImhhbW1pbmciLCAia3VsbGJhY2siLiAibWFuaGF0dGFuIiwgIm1heGltdW0iLCAiY2FuYmVycmEiLCAibWlua293c2tpIgpoZWFkKGRpc3Rhbnpfd29lcnRlcltvcmRlcihkaXN0YW56X3dvZXJ0ZXJbLDFdLCBkZWNyZWFzaW5nID0gVCksXSkKYGBgCgpgYGB7cn0KIyBQbG90dGVuIGRlciBLYXBpdGVsbsOkaGUgdmlhIEtvc2ludXPDpGhubGljaGtlaXQgKGhpZXIgZsO8ciBLYXBpdGVsIDEpCmFlaG5saWNoa2VpdF9rYXBpdGVsIDwtIGRhdGEuZnJhbWUoVGV4dCA9IGZhY3Rvcihrb3JwdXMuc3RhdHMkVGV4dCwgbGV2ZWxzID0gcmV2KGtvcnB1cy5zdGF0cyRUZXh0KSksIGFzLm1hdHJpeCh0ZXh0c3RhdF9zaW1pbChtZWluZS5kZm0sIHNlbGVjdGlvbiA9ICIwMSIsIG1hcmdpbiA9ICJkb2N1bWVudHMiLCBtZXRob2QgPSAiY29zaW5lIikpKQpnZ3Bsb3QoYWVobmxpY2hrZWl0X2thcGl0ZWwsIGFlcyhYMDEsIFRleHQpKSArIGdlb21fcG9pbnQoc2l6ZSA9IDIuNSkgKyBnZ3RpdGxlKCJLYXBpdGVsbsOkaGUgdmlhIEtvc2ludXPDpGhubGljaGtlaXQgKGhpZXIgZsO8ciBLYXBpdGVsIDEpIikgKyB4bGFiKCJLb3NpbnVuc8OkaG5saWNoa2VpdCIpCmBgYAoKYGBge3J9CiMgQmVyZWNobnVuZyB6YWhscmVpY2hlciBNZXRyaWtlbiBmw7xyIGRpZSBsZXhpa2FsaXNjaGUgRGl2ZXJzaXTDpHQgZGVzIEthcGl0ZWwgbWl0IHRleHRzdGF0X2xleGRpdigpCmxleGRpdmVyc2l0YWV0IDwtIHRleHRzdGF0X2xleGRpdihtZWluZS5kZm0pCmhlYWQobGV4ZGl2ZXJzaXRhZXQpCndyaXRlX2RlbGltKGxleGRpdmVyc2l0YWV0LCBwYXRoID0gImxleGRpdmVyc2l0YWV0LmNzdiIsIGRlbGltID0gIjsiKSAjIERhdGVpIGlzdCBFeGNlbC1rb21wYXRpYmVsCmBgYAoKYGBge3J9CiMgS2V5bmVzcyBuYWNoIFRleHQgKGhpZXIgZsO8ciBLYXBpdGVsIDAxLCAwMywgMjYsIDM0KQprZXluZXNzIDwtIHRleHRzdGF0X2tleW5lc3MobWVpbmUuZGZtLCB0YXJnZXQgPSAiMDEiLCBtZWFzdXJlID0gImxyIikKdGV4dHBsb3Rfa2V5bmVzcyhrZXluZXNzKQprZXluZXNzIDwtIHRleHRzdGF0X2tleW5lc3MobWVpbmUuZGZtLCB0YXJnZXQgPSAiMDMiLCBtZWFzdXJlID0gImxyIikKdGV4dHBsb3Rfa2V5bmVzcyhrZXluZXNzKQprZXluZXNzIDwtIHRleHRzdGF0X2tleW5lc3MobWVpbmUuZGZtLCB0YXJnZXQgPSAiMjYiLCBtZWFzdXJlID0gImxyIikKdGV4dHBsb3Rfa2V5bmVzcyhrZXluZXNzKQprZXluZXNzIDwtIHRleHRzdGF0X2tleW5lc3MobWVpbmUuZGZtLCB0YXJnZXQgPSAiMzQiLCBtZWFzdXJlID0gImxyIikKdGV4dHBsb3Rfa2V5bmVzcyhrZXluZXNzKQojIGFiZ2VzY2huaXR0ZW5lIGxhYmVscyBzaWNodGJhciBtYWNoZW46CiMgdGV4dHBsb3Rfa2V5bmVzcyhrZXluZXNzKSArIHNjYWxlX3lfY29udGludW91cyhsaW1pdHMgPSBjKC03NSwgMTMwKSkKYGBgCg==