Project: Text Summarizer

git clone <https://github.com/yourusername/advanced-text-summarizer.git>
cd advanced-text-summarizer

pip install -r requirements.txt

python summarizer.py --text "这里是需要摘要的长文本"

python summarizer.py --batch "文本 1" "文本 2" "文本 3"

from summarizer import AdvancedTextSummarizer

# 初始化摘要器
summarizer = AdvancedTextSummarizer()

# 需要摘要的文本
text = """
人工智能 (AI) 的发展对全球各个行业都产生了重大影响。
从医疗保健到金融，人工智能驱动的应用简化了操作，提高了准确性，
并开启了新的可能性。
"""

# 生成并打印摘要
summary_data = summarizer.summarize(text)
summarizer.print_summary(summary_data)

# 直接访问摘要
print("摘要:", summary_data["summary"])

from summarizer import AdvancedTextSummarizer, SummarizerConfig

# 自定义配置
config = SummarizerConfig(
    model_name="facebook/bart-large-cnn",
    quantize=True,           # 启用量化以提高速度
    max_length=100,          # 最大摘要长度
    min_length=30,           # 最小摘要长度
    repetition_penalty=2.0   # 更强的重复惩罚
)

# 使用上下文管理器进行资源管理
with AdvancedTextSummarizer(config) as summarizer:
    text = "这里是你的长文本..."
    summary_data = summarizer.summarize(text)
    summarizer.print_summary(summary_data)

import asyncio
from summarizer import AdvancedTextSummarizer

async def main():
    summarizer = AdvancedTextSummarizer()
    texts = [
        "人工智能正在通过更好的诊断彻底改变医疗保健。",
        "自动驾驶汽车使用机器学习进行导航。",
    ]

    # 摘要多个文本
    summaries = await summarizer.summarize_batch_async(texts)
    for text, summary in zip(texts, summaries):
        print(f"原文: {text}")
        print(f"摘要: {summary}\\\\n")

# 运行异步函数
asyncio.run(main())

╭────────────────────── 文本摘要 ─────────────────────╮
│                                                    │
│   原文文本    这里是你的输入文本...                    │
│   摘要          [粗体绿色]一个简洁的摘要...[/]         │
│                                                    │
╰──────────── 使用 BART 生成 ─────────────────────────╯

python summarizer.py  # 在 main 函数中取消注释 run_tests()

Text Summarizer 文本摘要器

A text summarization tool powered by BART (Bidirectional and Auto-Regressive Transformer) from Facebook, enhanced with advanced features like batch processing, quantization, and a rich console output.

Features

• Advanced Summarization: Uses BART-large-CNN model for high-quality summaries
• Text Preprocessing: Cleans and normalizes input text
• Long Text Handling: Splits and summarizes lengthy documents
• Batch Processing: Summarize multiple texts efficiently with async support
• Quantization: Optional model quantization for faster inference
• Rich Output: Beautiful console formatting using Rich library
• Configurable: Flexible parameters via configuration class
• Error Handling: Robust validation and logging
• Memory Management: Context manager support and cache clearing

Installation

git clone <https://github.com/yourusername/advanced-text-summarizer.git>
cd advanced-text-summarizer

Install dependencies:

pip install -r requirements.txt

Requirements

• Python 3.8+
• torch
• transformers
• rich
• See 'requirements.txt' for full list

Usage

Command Line

Summarize a single text:

python summarizer.py --text "Your long text here that needs summarization"

Summarize multiple texts:

python summarizer.py --batch "Text 1" "Text 2" "Text 3"

Python API

• The AdvancedTextSummarizer class provides a programmatic interface for text summarization, allowing us to integrate it into our Python scripts or applications.
• Below are examples of how to use it.

Basic Usage

Summarize a single text with default settings:

from summarizer import AdvancedTextSummarizer

# Initialize the summarizer
summarizer = AdvancedTextSummarizer()

# Text to summarize
text = """
The development of artificial intelligence (AI) has significantly impacted various industries worldwide.
From healthcare to finance, AI-powered applications have streamlined operations, improved accuracy,
and unlocked new possibilities.
"""

# Generate and print summary
summary_data = summarizer.summarize(text)
summarizer.print_summary(summary_data)

# Access the summary directly
print("Summary:", summary_data["summary"])

Custom Configuration

Use SummarizerConfig to customize the summarizer:

from summarizer import AdvancedTextSummarizer, SummarizerConfig

# Custom configuration
config = SummarizerConfig(
    model_name="facebook/bart-large-cnn",
    quantize=True,           # Enable quantization for speed
    max_length=100,          # Maximum summary length
    min_length=30,           # Minimum summary length
    repetition_penalty=2.0   # Stronger repetition penalty
)

# Use context manager for resource management
with AdvancedTextSummarizer(config) as summarizer:
    text = "Your long text here..."
    summary_data = summarizer.summarize(text)
    summarizer.print_summary(summary_data)

Batch Processing

Summarize multiple texts asynchronously:

import asyncio
from summarizer import AdvancedTextSummarizer

async def main():
    summarizer = AdvancedTextSummarizer()
    texts = [
        "AI is revolutionizing healthcare with better diagnostics.",
        "Self-driving cars use machine learning to navigate.",
    ]

    # Summarize multiple texts
    summaries = await summarizer.summarize_batch_async(texts)
    for text, summary in zip(texts, summaries):
        print(f"Original: {text}")
        print(f"Summary: {summary}\\n")

# Run the async function
asyncio.run(main())

Key Methods

• summarize(text, max_length=150, min_length=50, ...)
  Summarizes a single text. Returns a dictionary with original_text, cleaned_text, and summary.
• summarize_batch_async(texts, batch_size=2, ...)
  Asynchronously summarizes multiple texts. Returns a list of summaries.
• print_summary(summary_data)
  Displays a formatted summary using the Rich library.
• clear_cache()
  Clears the internal cache to free memory.

Notes

• Ensure dependencies are installed first: pip install -r requirements.txt
• Requires Python 3.8+ for asyncio support
• Use the context manager (with statement) for proper resource cleanup

Sample Text from Wikipedia

"Reinforcement learning (RL) is an interdisciplinary area of machine learning and optimal control concerned with how an intelligent agent should take actions in a dynamic environment in order to maximize a reward signal. Reinforcement learning is one of the three basic machine learning paradigms, alongside supervised learning and unsupervised learning.

Q-learning at its simplest stores data in tables. This approach becomes infeasible as the number of states/actions increases (e.g., if the state space or action space were continuous), as the probability of the agent visiting a particular state and performing a particular action diminishes.

Reinforcement learning differs from supervised learning in not needing labelled input-output pairs to be presented, and in not needing sub-optimal actions to be explicitly corrected. Instead, the focus is on finding a balance between exploration (of uncharted territory) and exploitation (of current knowledge) with the goal of maximizing the cumulative reward (the feedback of which might be incomplete or delayed).[1] The search for this balance is known as the exploration–exploitation dilemma.

The environment is typically stated in the form of a Markov decision process (MDP), as many reinforcement learning algorithms use dynamic programming techniques.[2] The main difference between classical dynamic programming methods and reinforcement learning algorithms is that the latter do not assume knowledge of an exact mathematical model of the Markov decision process, and they target large MDPs where exact methods become infeasible.[3]"

Example Output

╭────────────────────── Text Summary ──────────────────────╮
│                                                          │
│   Original Text    Your input text goes here...          │
│   Summary          [bold green]A concise summary...[/]   │
│                                                          │
╰──────────── Generated with BART ─────────────────────────╯

Configuration Options

Development

Running Tests

python summarizer.py  # Uncomment run_tests() in main

Contributing

• Fork the repository
• Create a feature branch (git checkout -b feature/amazing-feature)
• Commit your changes (git commit -am 'Add amazing feature')
• Push to the branch (git push origin feature/amazing-feature)
• Create a Pull Request

Acknowledgments

• Built with Transformers by Hugging Face
• Enhanced display with Rich
• Sample text from Wikipedia
• Inspired by Text Summarization with DistillBart Model

License

• Distributed under the GNU Affero General Public License v3.0 License. See LICENSE for more information.